[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:51:59 dist-test-slave-j2vl krb5kdc[10132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:51:59 dist-test-slave-j2vl krb5kdc[10132](info): set up 2 sockets
Aug 09 00:51:59 dist-test-slave-j2vl krb5kdc[10132](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 00:52:00.980677 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.443s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:52:01 dist-test-slave-j2vl krb5kdc[10132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:52:01.207275 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:37805
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:01.891887 10153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:01.892382 10153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:01.892616 10153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:01.903426 10153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:01.903702 10153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:01.903858 10153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:01.904035 10153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:01.904160 10153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:52:01.917732 10153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:37805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:37805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:01.919916 10153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:01.922155 10153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:01.949716 10165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:01.949880 10161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:01.952149 10163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:01.954030 10153 server_base.cc:1034] running on GCE node
I20260809 00:52:01.955503 10153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:01.958151 10153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:01.959594 10153 hybrid_clock.cc:648] HybridClock initialized: now 1786236721959410 us; error 150 us; skew 500 ppm
I20260809 00:52:01.960273 10153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:01 dist-test-slave-j2vl krb5kdc[10132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236721, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:01.965255 10153 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:52:01.968469 10153 webserver.cc:460] Webserver started at http://127.9.225.62:42605/ using document root <none> and password file <none>
I20260809 00:52:01.969864 10153 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:01.970058 10153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:01.970574 10153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:01.976956 10153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "07a4ce03dfa945fdb2bdd5af918b77c8"
format_stamp: "Formatted at 2026-08-09 00:52:01 on dist-test-slave-j2vl"
server_key: "825d535e55dddc504eba7c754a48a44b"
server_key_iv: "7ef7ff412c4b9020bf6cdfda82e1c123"
server_key_version: "encryptionkey@0"
I20260809 00:52:01.978204 10153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "07a4ce03dfa945fdb2bdd5af918b77c8"
format_stamp: "Formatted at 2026-08-09 00:52:01 on dist-test-slave-j2vl"
server_key: "825d535e55dddc504eba7c754a48a44b"
server_key_iv: "7ef7ff412c4b9020bf6cdfda82e1c123"
server_key_version: "encryptionkey@0"
I20260809 00:52:01.988353 10153 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260809 00:52:01.995857 10174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:01.998479 10153 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.002s
I20260809 00:52:01.998870 10153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "07a4ce03dfa945fdb2bdd5af918b77c8"
format_stamp: "Formatted at 2026-08-09 00:52:01 on dist-test-slave-j2vl"
server_key: "825d535e55dddc504eba7c754a48a44b"
server_key_iv: "7ef7ff412c4b9020bf6cdfda82e1c123"
server_key_version: "encryptionkey@0"
I20260809 00:52:01.999234 10153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:02.036515 10153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:02.043547 10153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:02.044067 10153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:02.095307 10153 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:37805
I20260809 00:52:02.095433 10247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:37805 every 8 connection(s)
I20260809 00:52:02.098096 10153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:02.099599 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 10153
I20260809 00:52:02.100207 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:02.103809 10116 external_mini_cluster.cc:1428] Setting key a87779747ff7f67a6490565f60628e61
I20260809 00:52:02.108091 10248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:02.134737 10248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8: Bootstrap starting.
I20260809 00:52:02.140954 10248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:02.143039 10248 log.cc:824] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:02.147679 10248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8: No bootstrap required, opened a new log
Aug 09 00:52:02 dist-test-slave-j2vl krb5kdc[10132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:02.158846 10248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a4ce03dfa945fdb2bdd5af918b77c8" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 37805 } }
I20260809 00:52:02.159586 10248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:02.159848 10248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a4ce03dfa945fdb2bdd5af918b77c8, State: Initialized, Role: FOLLOWER
I20260809 00:52:02.161664 10248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a4ce03dfa945fdb2bdd5af918b77c8" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 37805 } }
I20260809 00:52:02.162187 10248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:02.162528 10248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:02.162878 10248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:02.164654 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:02.111032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.9.225.62:37805)
0809 00:52:02.116565 (+ 5533us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:02.116578 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:02.116635 (+ 57us) server_negotiation.cc:407] Connection header received
0809 00:52:02.132862 (+ 16227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:02.132952 (+ 90us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:02.134553 (+ 1601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:02.135164 (+ 611us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:02.140077 (+ 4913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:02.141953 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:02.144204 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:02.145024 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:02.145329 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:02.152583 (+ 7254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:02.152655 (+ 72us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:02.152697 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:02.152757 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:02.156666 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:02.157943 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:02.157951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:02.157958 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:02.158191 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:02.158938 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:02.158947 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:02.158952 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:02.159966 (+ 1014us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:02.160446 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:02.162190 (+ 1744us) server_negotiation.cc:299] Negotiation successful
0809 00:52:02.162647 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5223,"thread_start_us":4796,"threads_started":1}
I20260809 00:52:02.166345 10248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a4ce03dfa945fdb2bdd5af918b77c8" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 37805 } }
I20260809 00:52:02.167398 10248 leader_election.cc:302] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07a4ce03dfa945fdb2bdd5af918b77c8; no voters:
I20260809 00:52:02.168511 10248 leader_election.cc:288] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:02.168910 10257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:02.172497 10248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:02.173501 10257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [term 1 LEADER]: Becoming Leader. State: Replica: 07a4ce03dfa945fdb2bdd5af918b77c8, State: Running, Role: LEADER
I20260809 00:52:02.174408 10257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a4ce03dfa945fdb2bdd5af918b77c8" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 37805 } }
I20260809 00:52:02.181471 10258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07a4ce03dfa945fdb2bdd5af918b77c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a4ce03dfa945fdb2bdd5af918b77c8" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 37805 } } }
I20260809 00:52:02.181504 10261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a4ce03dfa945fdb2bdd5af918b77c8. Latest consensus state: current_term: 1 leader_uuid: "07a4ce03dfa945fdb2bdd5af918b77c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a4ce03dfa945fdb2bdd5af918b77c8" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 37805 } } }
I20260809 00:52:02.182328 10261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:02.182353 10258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:02.186196 10273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:02.195736 10273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:02.211474 10273 catalog_manager.cc:1347] Generated new cluster ID: 895e78018a2c44608b707cf27910c5e6
I20260809 00:52:02.211679 10273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:02.228039 10273 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:02.229820 10273 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:02.241410 10273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07a4ce03dfa945fdb2bdd5af918b77c8: Generated new TSK 0
I20260809 00:52:02.242967 10273 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:52:02.305820 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:37805
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:02.907974 10289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:02.908427 10289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:02.908721 10289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:02.919492 10289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:02.919726 10289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:02.919981 10289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:52:02.934064 10289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:37805
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:02.935958 10289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:02.937891 10289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:02.955619 10296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:02.955817 10297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:02.956802 10299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:02.957701 10289 server_base.cc:1034] running on GCE node
I20260809 00:52:02.958606 10289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:02.959913 10289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:02.961202 10289 hybrid_clock.cc:648] HybridClock initialized: now 1786236722961082 us; error 68 us; skew 500 ppm
I20260809 00:52:02.961714 10289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:02 dist-test-slave-j2vl krb5kdc[10132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236722, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:02.966915 10289 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:52:02.969319 10289 webserver.cc:460] Webserver started at http://127.9.225.1:46481/ using document root <none> and password file <none>
I20260809 00:52:02.970448 10289 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:02.970686 10289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:02.971185 10289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:02.975994 10289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "a36ec57f7aaa47c286c0100dc777d65b"
format_stamp: "Formatted at 2026-08-09 00:52:02 on dist-test-slave-j2vl"
server_key: "e9a729f3eae0f160e8db731c337324cd"
server_key_iv: "32035a4a3e3096118695da641c9bb867"
server_key_version: "encryptionkey@0"
I20260809 00:52:02.977126 10289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "a36ec57f7aaa47c286c0100dc777d65b"
format_stamp: "Formatted at 2026-08-09 00:52:02 on dist-test-slave-j2vl"
server_key: "e9a729f3eae0f160e8db731c337324cd"
server_key_iv: "32035a4a3e3096118695da641c9bb867"
server_key_version: "encryptionkey@0"
I20260809 00:52:02.985119 10289 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.006s
I20260809 00:52:02.991050 10314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:02.992838 10289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:52:02.993120 10289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "a36ec57f7aaa47c286c0100dc777d65b"
format_stamp: "Formatted at 2026-08-09 00:52:02 on dist-test-slave-j2vl"
server_key: "e9a729f3eae0f160e8db731c337324cd"
server_key_iv: "32035a4a3e3096118695da641c9bb867"
server_key_version: "encryptionkey@0"
I20260809 00:52:02.993495 10289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:03.014163 10289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:03.020517 10289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:03.021062 10289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:03.023075 10289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:03.025724 10289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:03.025929 10289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:03.026247 10289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:03.026423 10289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:03.130955 10289 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:37505
I20260809 00:52:03.131029 10517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:37505 every 8 connection(s)
I20260809 00:52:03.133203 10289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:52:03.140689 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 10289
I20260809 00:52:03.141112 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:52:03.142442 10116 external_mini_cluster.cc:1428] Setting key c38d03d9c0cadb4ac2f1593619590ee7
Aug 09 00:52:03 dist-test-slave-j2vl krb5kdc[10132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236722, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
I20260809 00:52:03.168094 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:03.138062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:40217 (local address 127.9.225.62:37805)
0809 00:52:03.139112 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:03.139120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:03.140555 (+ 1435us) server_negotiation.cc:407] Connection header received
0809 00:52:03.144781 (+ 4226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:03.144788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:03.144882 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:03.145026 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:03.148438 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:03.149626 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:03.152887 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:03.153613 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:03.153780 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:03.159603 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:03.159610 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:03.159613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:03.159655 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:03.163496 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:03.164444 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:03.164449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:03.164452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:03.164649 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:03.165300 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:03.165307 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:03.165311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:03.165799 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:03.166015 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:03.167054 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0809 00:52:03.167325 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":678,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
I20260809 00:52:03.169862 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:03.138665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:37805 (local address 127.9.225.1:40217)
0809 00:52:03.140329 (+ 1664us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:03.140390 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:03.144470 (+ 4080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:03.145421 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:03.145435 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:03.146613 (+ 1178us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:03.148134 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:03.148169 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:03.149796 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:03.149805 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:03.152701 (+ 2896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:03.152719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:03.154023 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:03.154034 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:03.154395 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:03.155346 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:03.155372 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:03.159205 (+ 3833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:03.163686 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:03.163698 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:03.163720 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:03.164284 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:03.164793 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:03.164798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:03.164801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:03.165139 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:03.165974 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:03.165983 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:03.166545 (+ 562us) client_negotiation.cc:769] Sending connection context
0809 00:52:03.166948 (+ 403us) client_negotiation.cc:240] Negotiation successful
0809 00:52:03.167780 (+ 832us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"thread_start_us":788,"threads_started":1}
I20260809 00:52:03.171854 10518 heartbeater.cc:347] Connected to a master server at 127.9.225.62:37805
I20260809 00:52:03.172359 10518 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:03.173456 10518 heartbeater.cc:511] Master 127.9.225.62:37805 requested a full tablet report, sending...
I20260809 00:52:03.178093 10200 ts_manager.cc:194] Registered new tserver with Master: a36ec57f7aaa47c286c0100dc777d65b (127.9.225.1:37505)
I20260809 00:52:03.180490 10200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:40217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:52:03.198323 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:37805
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:03.843736 10526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:03.844159 10526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:03.844329 10526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:03.854399 10526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:03.854638 10526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:03.854887 10526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:52:03.867915 10526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:37805
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:03.869478 10526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:03.871153 10526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:03.890756 10535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:03.891996 10540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:03.892076 10538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:03.892952 10526 server_base.cc:1034] running on GCE node
I20260809 00:52:03.893968 10526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:03.895227 10526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:03.896518 10526 hybrid_clock.cc:648] HybridClock initialized: now 1786236723896428 us; error 58 us; skew 500 ppm
I20260809 00:52:03.896935 10526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:03 dist-test-slave-j2vl krb5kdc[10132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236723, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:03.901638 10526 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:52:03.904054 10526 webserver.cc:460] Webserver started at http://127.9.225.2:34045/ using document root <none> and password file <none>
I20260809 00:52:03.905056 10526 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:03.905297 10526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:03.905831 10526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:03.909977 10526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "1ff8ab39356c461c8379dfe1b48f42bb"
format_stamp: "Formatted at 2026-08-09 00:52:03 on dist-test-slave-j2vl"
server_key: "ff9b872aa9c5944e5667cc863b77a209"
server_key_iv: "7a57d569eb719e80d52c2d311f46108a"
server_key_version: "encryptionkey@0"
I20260809 00:52:03.910903 10526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "1ff8ab39356c461c8379dfe1b48f42bb"
format_stamp: "Formatted at 2026-08-09 00:52:03 on dist-test-slave-j2vl"
server_key: "ff9b872aa9c5944e5667cc863b77a209"
server_key_iv: "7a57d569eb719e80d52c2d311f46108a"
server_key_version: "encryptionkey@0"
I20260809 00:52:03.918084 10526 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260809 00:52:03.923235 10555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:03.924899 10526 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260809 00:52:03.925184 10526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "1ff8ab39356c461c8379dfe1b48f42bb"
format_stamp: "Formatted at 2026-08-09 00:52:03 on dist-test-slave-j2vl"
server_key: "ff9b872aa9c5944e5667cc863b77a209"
server_key_iv: "7a57d569eb719e80d52c2d311f46108a"
server_key_version: "encryptionkey@0"
I20260809 00:52:03.925546 10526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:03.944113 10526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:03.949710 10526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:03.950199 10526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:03.952008 10526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:03.954129 10526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:03.954306 10526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:03.954540 10526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:03.954636 10526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:04.030511 10526 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:45557
I20260809 00:52:04.030560 10739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:45557 every 8 connection(s)
I20260809 00:52:04.032274 10526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:52:04.033393 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 10526
I20260809 00:52:04.033681 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:52:04.034582 10116 external_mini_cluster.cc:1428] Setting key d5b1ad0083efbe647c4de6ac115d8823
Aug 09 00:52:04 dist-test-slave-j2vl krb5kdc[10132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236723, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:04.063980 10743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:04.037024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:38493 (local address 127.9.225.62:37805)
0809 00:52:04.037924 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:04.037930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:04.038943 (+ 1013us) server_negotiation.cc:407] Connection header received
0809 00:52:04.041776 (+ 2833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:04.041785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:04.041947 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:04.042284 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:04.044397 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:04.045967 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:04.048277 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:04.049402 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:04.049567 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:04.055814 (+ 6247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:04.055820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:04.055823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:04.055864 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:04.059339 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:04.060740 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:04.060754 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:04.060756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:04.060892 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:04.061704 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:04.061708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:04.061711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:04.062041 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:04.062167 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:04.063399 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0809 00:52:04.063623 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":450,"threads_started":1}
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
I20260809 00:52:04.065461 10745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:04.037115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:37805 (local address 127.9.225.2:38493)
0809 00:52:04.038723 (+ 1608us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:04.038777 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:04.041402 (+ 2625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:04.042596 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:04.042606 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:04.043376 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:04.044166 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:04.044187 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:04.046242 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:04.046247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:04.048088 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:04.048104 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:04.049729 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:04.049739 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:04.050075 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:04.051052 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:04.051127 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:04.055441 (+ 4314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:04.059540 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:04.059555 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:04.059582 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:04.060584 (+ 1002us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:04.061078 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:04.061086 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:04.061092 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:04.061551 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:04.062220 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:04.062230 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:04.062844 (+ 614us) client_negotiation.cc:769] Sending connection context
0809 00:52:04.063261 (+ 417us) client_negotiation.cc:240] Negotiation successful
0809 00:52:04.064007 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1192,"thread_start_us":801,"threads_started":1}
I20260809 00:52:04.067268 10740 heartbeater.cc:347] Connected to a master server at 127.9.225.62:37805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
I20260809 00:52:04.067723 10740 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:04.068907 10740 heartbeater.cc:511] Master 127.9.225.62:37805 requested a full tablet report, sending...
I20260809 00:52:04.071239 10200 ts_manager.cc:194] Registered new tserver with Master: 1ff8ab39356c461c8379dfe1b48f42bb (127.9.225.2:45557)
I20260809 00:52:04.073026 10200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:38493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:52:04.100963 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:37805
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:52:04.184440 10518 heartbeater.cc:503] Master 127.9.225.62:37805 was elected leader, sending a full tablet report...
W20260809 00:52:04.703436 10751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:04.703986 10751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:04.704244 10751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:04.714892 10751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:04.715131 10751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:04.715395 10751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:52:04.728762 10751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:37805
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:04.730672 10751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:04.732647 10751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:04.752545 10767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:04.753769 10762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:04.754215 10761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:04.761008 10751 server_base.cc:1034] running on GCE node
I20260809 00:52:04.762574 10751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:04.765048 10751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:04.766570 10751 hybrid_clock.cc:648] HybridClock initialized: now 1786236724766419 us; error 74 us; skew 500 ppm
I20260809 00:52:04.767185 10751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:04 dist-test-slave-j2vl krb5kdc[10132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236724, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:04.778788 10751 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:52:04.782068 10751 webserver.cc:460] Webserver started at http://127.9.225.3:33653/ using document root <none> and password file <none>
I20260809 00:52:04.783051 10751 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:04.783222 10751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:04.783667 10751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:04.787673 10751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "b8d9c37c8a344d71bc78ec3446d7aeaa"
format_stamp: "Formatted at 2026-08-09 00:52:04 on dist-test-slave-j2vl"
server_key: "9f9d55d751ebed94aa802f191d2d8eb0"
server_key_iv: "1269f13aa5b9a67cafc7e08a9045ab7a"
server_key_version: "encryptionkey@0"
I20260809 00:52:04.788581 10751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "b8d9c37c8a344d71bc78ec3446d7aeaa"
format_stamp: "Formatted at 2026-08-09 00:52:04 on dist-test-slave-j2vl"
server_key: "9f9d55d751ebed94aa802f191d2d8eb0"
server_key_iv: "1269f13aa5b9a67cafc7e08a9045ab7a"
server_key_version: "encryptionkey@0"
I20260809 00:52:04.795468 10751 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20260809 00:52:04.800601 10777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:04.802336 10751 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260809 00:52:04.802552 10751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "b8d9c37c8a344d71bc78ec3446d7aeaa"
format_stamp: "Formatted at 2026-08-09 00:52:04 on dist-test-slave-j2vl"
server_key: "9f9d55d751ebed94aa802f191d2d8eb0"
server_key_iv: "1269f13aa5b9a67cafc7e08a9045ab7a"
server_key_version: "encryptionkey@0"
I20260809 00:52:04.802806 10751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:04.828955 10751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:04.834437 10751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:04.834827 10751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:04.836340 10751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:04.838305 10751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:04.838477 10751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:04.838641 10751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:04.838721 10751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:04.916865 10751 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:34761
I20260809 00:52:04.917148 10962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:34761 every 8 connection(s)
I20260809 00:52:04.918601 10751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:52:04.922027 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 10751
I20260809 00:52:04.922288 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:52:04.922999 10116 external_mini_cluster.cc:1428] Setting key b5b77ffd7bc1c7be80aa05333707a49a
Aug 09 00:52:04 dist-test-slave-j2vl krb5kdc[10132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236724, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:04.947326 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:04.922519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:56601 (local address 127.9.225.62:37805)
0809 00:52:04.923147 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:04.923156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:04.925421 (+ 2265us) server_negotiation.cc:407] Connection header received
0809 00:52:04.928450 (+ 3029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:04.928458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:04.928551 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:04.928776 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:04.930913 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:04.932093 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:04.934098 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:04.934614 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:04.934730 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:04.939018 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:04.939023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:04.939026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:04.939089 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:04.943576 (+ 4487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:04.944407 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:04.944413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:04.944419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:04.944572 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:04.945171 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:04.945176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:04.945178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:04.945539 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:04.945721 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:04.946559 (+ 838us) server_negotiation.cc:299] Negotiation successful
0809 00:52:04.946818 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":300,"threads_started":1}
I20260809 00:52:04.948549 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:04.923363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:37805 (local address 127.9.225.3:56601)
0809 00:52:04.924942 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:04.925019 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:04.928110 (+ 3091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:04.929038 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:04.929047 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:04.929886 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:04.930735 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:04.930758 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:04.932276 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:04.932279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:04.933966 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:04.933980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:04.934874 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:04.934879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:04.935162 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:04.935893 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:04.935918 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:04.938696 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:04.943728 (+ 5032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:04.943738 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:04.943757 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:04.944261 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:04.944696 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:04.944701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:04.944705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:04.945031 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:04.945675 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:04.945684 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:04.946118 (+ 434us) client_negotiation.cc:769] Sending connection context
0809 00:52:04.946431 (+ 313us) client_negotiation.cc:240] Negotiation successful
0809 00:52:04.947097 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"thread_start_us":726,"threads_started":1}
I20260809 00:52:04.950455 10963 heartbeater.cc:347] Connected to a master server at 127.9.225.62:37805
I20260809 00:52:04.950831 10963 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:04.951740 10963 heartbeater.cc:511] Master 127.9.225.62:37805 requested a full tablet report, sending...
I20260809 00:52:04.953657 10200 ts_manager.cc:194] Registered new tserver with Master: b8d9c37c8a344d71bc78ec3446d7aeaa (127.9.225.3:34761)
I20260809 00:52:04.955188 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:52:04.955188 10200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:56601
Aug 09 00:52:04 dist-test-slave-j2vl krb5kdc[10132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236724, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 00:52:05 dist-test-slave-j2vl krb5kdc[10132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236724, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:05.022305 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:05.004410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.9.225.1:37505)
0809 00:52:05.005555 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:05.005562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:05.005592 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:52:05.005656 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:05.005663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:05.005938 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:05.006107 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:05.007836 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:05.009033 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:05.010453 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:05.011098 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:05.011211 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:05.015977 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:05.015996 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:05.016017 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:05.016097 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:05.019516 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:05.020263 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:05.020272 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:05.020274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:05.020448 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:05.020814 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:05.020817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:05.020819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:05.021186 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:05.021525 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:05.021770 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 00:52:05.021998 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":820,"threads_started":1}
W20260809 00:52:05.023871 10361 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58020
Aug 09 00:52:05 dist-test-slave-j2vl krb5kdc[10132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260809 00:52:05.076711 10740 heartbeater.cc:503] Master 127.9.225.62:37805 was elected leader, sending a full tablet report...
Aug 09 00:52:05 dist-test-slave-j2vl krb5kdc[10132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:05.085988 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:05.070087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.9.225.1:37505)
0809 00:52:05.070358 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:05.070364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:05.070590 (+ 226us) server_negotiation.cc:407] Connection header received
0809 00:52:05.070763 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:05.070767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:05.070838 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:05.070934 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:05.072390 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:05.073096 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:05.074458 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:05.074883 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:05.074987 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:05.080420 (+ 5433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:05.080426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:05.080430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:05.080482 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:05.083319 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:05.084078 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:05.084083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:05.084085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:05.084230 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:05.084666 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:05.084669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:05.084671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:05.084942 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:05.085143 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:05.085529 (+ 386us) server_negotiation.cc:299] Negotiation successful
0809 00:52:05.085717 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260809 00:52:05.088354 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 10289
I20260809 00:52:05.103925 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:05.089543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.9.225.1:37505)
0809 00:52:05.089712 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:05.089716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:05.090162 (+ 446us) server_negotiation.cc:407] Connection header received
0809 00:52:05.090291 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:05.090295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:05.090354 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:05.090440 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:05.091630 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:05.092416 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:05.093775 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:05.094483 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:05.094625 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:05.097661 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:05.097667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:05.097670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:05.097707 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:05.100956 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:05.101915 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:05.101920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:05.101922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:05.102066 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:05.102513 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:05.102517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:05.102520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:05.102861 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:05.103055 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:05.103432 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 00:52:05.103612 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 00:52:05.104926 10509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:05.196611 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 10289
W20260809 00:52:05.223534 10182 connection.cc:535] server connection from 127.9.225.1:40217 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 00:52:05.223765 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 10526
Aug 09 00:52:05 dist-test-slave-j2vl krb5kdc[10132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:52:05.240935 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:05.224802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.9.225.2:45557)
0809 00:52:05.225527 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:05.225536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:05.225563 (+ 27us) server_negotiation.cc:407] Connection header received
0809 00:52:05.225629 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:05.225637 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:05.225859 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:05.226000 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:05.227239 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:05.228251 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:05.229699 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:05.230344 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:05.230455 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:05.234870 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:05.234884 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:05.234918 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:05.234961 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:05.238027 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:05.238748 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:05.238756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:05.238758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:05.238893 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:05.239268 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:05.239271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:05.239273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:05.239641 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:05.239879 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:05.240260 (+ 381us) server_negotiation.cc:299] Negotiation successful
0809 00:52:05.240490 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":304,"threads_started":1}
I20260809 00:52:05.242424 10734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:05.328724 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 10526
I20260809 00:52:05.358275 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 10751
Aug 09 00:52:05 dist-test-slave-j2vl krb5kdc[10132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:52:05.375505 10998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:05.359343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.9.225.3:34761)
0809 00:52:05.359999 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:05.360008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:05.360041 (+ 33us) server_negotiation.cc:407] Connection header received
0809 00:52:05.360112 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:05.360119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:05.360335 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:05.360472 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:05.361771 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:05.362775 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:05.364036 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:05.364600 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:05.364699 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:05.369216 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:05.369232 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:05.369251 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:05.369291 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:05.372454 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:05.373302 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:05.373311 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:05.373313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:05.373544 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:05.373926 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:05.373929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:05.373931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:05.374313 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:05.374555 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:05.374967 (+ 412us) server_negotiation.cc:299] Negotiation successful
0809 00:52:05.375187 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":283,"threads_started":1}
I20260809 00:52:05.376904 10954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:05.462886 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 10751
I20260809 00:52:05.487875 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 10153
I20260809 00:52:05.489292 10242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:05.599279 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 10153
2026-08-09T00:52:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6140 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:52:05 dist-test-slave-j2vl krb5kdc[11005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:52:05 dist-test-slave-j2vl krb5kdc[11005](info): set up 2 sockets
Aug 09 00:52:05 dist-test-slave-j2vl krb5kdc[11005](info): commencing operation
krb5kdc: starting...
W20260809 00:52:06.953073 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.304s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:52:07 dist-test-slave-j2vl krb5kdc[11005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:52:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:52:07.172698 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43621
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:43621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:07.524897 11029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:07.525251 11029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:07.525434 11029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:07.533033 11029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:07.533179 11029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:07.533275 11029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:07.533389 11029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:07.533450 11029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:52:07.544015 11029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:43621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:07.545651 11029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:07.547101 11029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:07.559669 11042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:07.559792 11037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:07.559826 11038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:07.560947 11029 server_base.cc:1034] running on GCE node
I20260809 00:52:07.561950 11029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:07.563601 11029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:07.564860 11029 hybrid_clock.cc:648] HybridClock initialized: now 1786236727564814 us; error 69 us; skew 500 ppm
I20260809 00:52:07.565275 11029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:07 dist-test-slave-j2vl krb5kdc[11005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236727, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:07.568691 11029 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:52:07.570489 11029 webserver.cc:460] Webserver started at http://127.9.225.62:32839/ using document root <none> and password file <none>
I20260809 00:52:07.571252 11029 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:07.571362 11029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:07.571659 11029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:07.575012 11029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "d0072577dff047c3a9e1fdfab99ac1be"
format_stamp: "Formatted at 2026-08-09 00:52:07 on dist-test-slave-j2vl"
server_key: "79ac014b6844607df27b00087d8acc8e"
server_key_iv: "56f3b82482bde317203af051f55b9f48"
server_key_version: "encryptionkey@0"
I20260809 00:52:07.575796 11029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "d0072577dff047c3a9e1fdfab99ac1be"
format_stamp: "Formatted at 2026-08-09 00:52:07 on dist-test-slave-j2vl"
server_key: "79ac014b6844607df27b00087d8acc8e"
server_key_iv: "56f3b82482bde317203af051f55b9f48"
server_key_version: "encryptionkey@0"
I20260809 00:52:07.581422 11029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 00:52:07.585136 11052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:07.586725 11029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 00:52:07.586898 11029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "d0072577dff047c3a9e1fdfab99ac1be"
format_stamp: "Formatted at 2026-08-09 00:52:07 on dist-test-slave-j2vl"
server_key: "79ac014b6844607df27b00087d8acc8e"
server_key_iv: "56f3b82482bde317203af051f55b9f48"
server_key_version: "encryptionkey@0"
I20260809 00:52:07.587093 11029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:07.612875 11029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:07.617285 11029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:07.617588 11029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:07.637799 11029 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:43621
I20260809 00:52:07.637842 11142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:43621 every 8 connection(s)
I20260809 00:52:07.639356 11029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:07.643954 11143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:07.646234 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 11029
I20260809 00:52:07.646490 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:07.647179 10116 external_mini_cluster.cc:1428] Setting key 53862b61426e4a57d8512a2257a0e6a4
I20260809 00:52:07.658999 11143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be: Bootstrap starting.
I20260809 00:52:07.662954 11143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be: Neither blocks nor log segments found. Creating new log.
Aug 09 00:52:07 dist-test-slave-j2vl krb5kdc[11005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:07.664140 11143 log.cc:824] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:07.668196 11143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be: No bootstrap required, opened a new log
I20260809 00:52:07.674072 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:07.649533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.9.225.62:43621)
0809 00:52:07.650486 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:07.650503 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:07.650561 (+ 58us) server_negotiation.cc:407] Connection header received
0809 00:52:07.653131 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:07.653165 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:07.653745 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:07.654224 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:07.655702 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:07.657858 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:07.659140 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:07.660205 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:07.660396 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:07.664647 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:07.664659 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:07.664686 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:07.664776 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:07.668935 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:07.669819 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:07.669828 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:07.669838 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:07.670078 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:07.670462 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:07.670467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:07.670471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:07.671262 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:07.671563 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:07.672043 (+ 480us) server_negotiation.cc:299] Negotiation successful
0809 00:52:07.672544 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":371,"threads_started":1}
I20260809 00:52:07.678646 11143 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0072577dff047c3a9e1fdfab99ac1be" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43621 } }
I20260809 00:52:07.679013 11143 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:07.679090 11143 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0072577dff047c3a9e1fdfab99ac1be, State: Initialized, Role: FOLLOWER
I20260809 00:52:07.679672 11143 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0072577dff047c3a9e1fdfab99ac1be" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43621 } }
I20260809 00:52:07.679946 11143 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:07.680095 11143 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:07.680289 11143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:07.682699 11143 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0072577dff047c3a9e1fdfab99ac1be" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43621 } }
I20260809 00:52:07.683292 11143 leader_election.cc:302] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0072577dff047c3a9e1fdfab99ac1be; no voters:
I20260809 00:52:07.683909 11143 leader_election.cc:288] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:07.684165 11152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:07.684882 11152 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [term 1 LEADER]: Becoming Leader. State: Replica: d0072577dff047c3a9e1fdfab99ac1be, State: Running, Role: LEADER
I20260809 00:52:07.685607 11152 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0072577dff047c3a9e1fdfab99ac1be" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43621 } }
I20260809 00:52:07.686528 11143 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:07.689191 11155 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0072577dff047c3a9e1fdfab99ac1be. Latest consensus state: current_term: 1 leader_uuid: "d0072577dff047c3a9e1fdfab99ac1be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0072577dff047c3a9e1fdfab99ac1be" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43621 } } }
I20260809 00:52:07.689776 11155 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:07.689181 11153 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0072577dff047c3a9e1fdfab99ac1be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0072577dff047c3a9e1fdfab99ac1be" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43621 } } }
I20260809 00:52:07.690174 11153 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:07.696548 11167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:07.703055 11167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:07.713570 11167 catalog_manager.cc:1347] Generated new cluster ID: 31b52fb8559b4297bd1133c73bb85925
I20260809 00:52:07.713717 11167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:07.729461 11167 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:07.730993 11167 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:07.744040 11167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0072577dff047c3a9e1fdfab99ac1be: Generated new TSK 0
I20260809 00:52:07.745044 11167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:52:07.810212 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43621
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:08.172642 11188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:08.173079 11188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:08.173255 11188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:08.180954 11188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:08.181116 11188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:08.181396 11188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:52:08.194777 11188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43621
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:08.197129 11188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:08.199124 11188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:08.213814 11201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:08.213860 11205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:08.214550 11200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:08.214570 11188 server_base.cc:1034] running on GCE node
I20260809 00:52:08.215355 11188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:08.216382 11188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:08.217618 11188 hybrid_clock.cc:648] HybridClock initialized: now 1786236728217567 us; error 44 us; skew 500 ppm
I20260809 00:52:08.217968 11188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:08 dist-test-slave-j2vl krb5kdc[11005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236728, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:08.221251 11188 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:52:08.223003 11188 webserver.cc:460] Webserver started at http://127.9.225.1:36783/ using document root <none> and password file <none>
I20260809 00:52:08.223730 11188 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:08.223842 11188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:08.224130 11188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:08.227286 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "9015ad79e31c4deb8f70c6ad695db55f"
format_stamp: "Formatted at 2026-08-09 00:52:08 on dist-test-slave-j2vl"
server_key: "eb98b928654c43939baeafdb77bb334e"
server_key_iv: "a121849f7f2bd41e9e84b377f044f17a"
server_key_version: "encryptionkey@0"
I20260809 00:52:08.228089 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "9015ad79e31c4deb8f70c6ad695db55f"
format_stamp: "Formatted at 2026-08-09 00:52:08 on dist-test-slave-j2vl"
server_key: "eb98b928654c43939baeafdb77bb334e"
server_key_iv: "a121849f7f2bd41e9e84b377f044f17a"
server_key_version: "encryptionkey@0"
I20260809 00:52:08.233444 11188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 00:52:08.237222 11218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:08.239157 11188 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:52:08.239354 11188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "9015ad79e31c4deb8f70c6ad695db55f"
format_stamp: "Formatted at 2026-08-09 00:52:08 on dist-test-slave-j2vl"
server_key: "eb98b928654c43939baeafdb77bb334e"
server_key_iv: "a121849f7f2bd41e9e84b377f044f17a"
server_key_version: "encryptionkey@0"
I20260809 00:52:08.239575 11188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:08.262997 11188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:08.267949 11188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:08.268226 11188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:08.269268 11188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:08.270738 11188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:08.270821 11188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:08.270921 11188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:08.270972 11188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:08.307749 11188 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:33891
I20260809 00:52:08.307837 11412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:33891 every 8 connection(s)
I20260809 00:52:08.309245 11188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:52:08.314481 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 11188
I20260809 00:52:08.314711 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:52:08.315541 10116 external_mini_cluster.cc:1428] Setting key c1b293024f6669b9b18485f15d911964
Aug 09 00:52:08 dist-test-slave-j2vl krb5kdc[11005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236728, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:08.333417 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:08.312019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:54367 (local address 127.9.225.62:43621)
0809 00:52:08.312500 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:08.312505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:08.313379 (+ 874us) server_negotiation.cc:407] Connection header received
0809 00:52:08.316085 (+ 2706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:08.316090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:08.316192 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:08.316328 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:08.318287 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:08.319022 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:08.321726 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:08.322156 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:08.322262 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:08.327021 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:08.327026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:08.327030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:08.327066 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:08.329776 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:08.330776 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:08.330780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:08.330782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:08.330929 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:08.331439 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:08.331442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:08.331444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:08.331726 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:08.331840 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:08.332892 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0809 00:52:08.333090 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":269,"threads_started":1}
I20260809 00:52:08.335256 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:08.312450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43621 (local address 127.9.225.1:54367)
0809 00:52:08.313174 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:08.313222 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:08.315751 (+ 2529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:08.316618 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:08.316632 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:08.317201 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:08.318095 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:08.318130 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:08.319226 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:08.319235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:08.321550 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:08.321594 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:08.322410 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:08.322427 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:08.322779 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:08.323579 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:08.323615 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:08.326734 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:08.329936 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:08.329948 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:08.329971 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:08.330638 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:08.331048 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:08.331055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:08.331062 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:08.331321 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:08.331864 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:08.331875 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:08.332419 (+ 544us) client_negotiation.cc:769] Sending connection context
0809 00:52:08.332792 (+ 373us) client_negotiation.cc:240] Negotiation successful
0809 00:52:08.333494 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":282,"threads_started":1}
I20260809 00:52:08.337294 11414 heartbeater.cc:347] Connected to a master server at 127.9.225.62:43621
I20260809 00:52:08.337657 11414 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
I20260809 00:52:08.338465 11414 heartbeater.cc:511] Master 127.9.225.62:43621 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
I20260809 00:52:08.340852 11081 ts_manager.cc:194] Registered new tserver with Master: 9015ad79e31c4deb8f70c6ad695db55f (127.9.225.1:33891)
I20260809 00:52:08.342469 11081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:54367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:52:08.366657 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43621
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:08.713830 11427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:08.714143 11427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:08.714244 11427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:08.721621 11427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:08.721746 11427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:08.721868 11427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:52:08.732035 11427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43621
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:08.733457 11427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:08.734875 11427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:08.748466 11438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:08.748588 11440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:08.748890 11443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:08.750780 11427 server_base.cc:1034] running on GCE node
I20260809 00:52:08.751398 11427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:08.752413 11427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:08.753660 11427 hybrid_clock.cc:648] HybridClock initialized: now 1786236728753607 us; error 50 us; skew 500 ppm
I20260809 00:52:08.754014 11427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:08 dist-test-slave-j2vl krb5kdc[11005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236728, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:08.757570 11427 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:52:08.759233 11427 webserver.cc:460] Webserver started at http://127.9.225.2:41813/ using document root <none> and password file <none>
I20260809 00:52:08.760033 11427 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:08.760195 11427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:08.760551 11427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:08.763751 11427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "e70c879491044bd08444023659b64479"
format_stamp: "Formatted at 2026-08-09 00:52:08 on dist-test-slave-j2vl"
server_key: "d4b37ac5ea250a2117b5584f6db81ad9"
server_key_iv: "2595b45ae1473d9374af4e2cd619be6d"
server_key_version: "encryptionkey@0"
I20260809 00:52:08.764650 11427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "e70c879491044bd08444023659b64479"
format_stamp: "Formatted at 2026-08-09 00:52:08 on dist-test-slave-j2vl"
server_key: "d4b37ac5ea250a2117b5584f6db81ad9"
server_key_iv: "2595b45ae1473d9374af4e2cd619be6d"
server_key_version: "encryptionkey@0"
I20260809 00:52:08.770202 11427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 00:52:08.773957 11452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:08.775434 11427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 00:52:08.775635 11427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "e70c879491044bd08444023659b64479"
format_stamp: "Formatted at 2026-08-09 00:52:08 on dist-test-slave-j2vl"
server_key: "d4b37ac5ea250a2117b5584f6db81ad9"
server_key_iv: "2595b45ae1473d9374af4e2cd619be6d"
server_key_version: "encryptionkey@0"
I20260809 00:52:08.775872 11427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:08.793627 11427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:08.798768 11427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:08.799113 11427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:08.800223 11427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:08.801771 11427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:08.801887 11427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:08.802054 11427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:08.802135 11427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:08.838179 11427 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:37557
I20260809 00:52:08.838199 11644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:37557 every 8 connection(s)
I20260809 00:52:08.839622 11427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:52:08.848685 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 11427
I20260809 00:52:08.848883 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:52:08.849444 10116 external_mini_cluster.cc:1428] Setting key fe9950efc00f200b3d9f7265479230f3
Aug 09 00:52:08 dist-test-slave-j2vl krb5kdc[11005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236728, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:08.864709 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:08.842417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:53745 (local address 127.9.225.62:43621)
0809 00:52:08.842888 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:08.842895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:08.844134 (+ 1239us) server_negotiation.cc:407] Connection header received
0809 00:52:08.846238 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:08.846243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:08.846317 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:08.846423 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:08.848000 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:08.848883 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:08.851094 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:08.851691 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:08.851854 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:08.856083 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:08.856099 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:08.856106 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:08.856196 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:08.860874 (+ 4678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:08.861972 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:08.861980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:08.861983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:08.862209 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:08.862731 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:08.862735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:08.862737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:08.863034 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:08.863152 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:08.864246 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0809 00:52:08.864442 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":211,"threads_started":1}
I20260809 00:52:08.866544 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:08.843035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43621 (local address 127.9.225.2:53745)
0809 00:52:08.843947 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:08.843991 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:08.845875 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:08.846688 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:08.846696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:08.847239 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:08.847814 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:08.847832 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:08.849120 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:08.849124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:08.850949 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:08.850965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:08.851926 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:08.851931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:08.852144 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:08.852884 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:08.852909 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:08.855655 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:08.861175 (+ 5520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:08.861184 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:08.861201 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:08.861786 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:08.862348 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:08.862351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:08.862353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:08.862598 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:08.863219 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:08.863231 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:08.863777 (+ 546us) client_negotiation.cc:769] Sending connection context
0809 00:52:08.864152 (+ 375us) client_negotiation.cc:240] Negotiation successful
0809 00:52:08.864854 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":373,"threads_started":1}
I20260809 00:52:08.868212 11646 heartbeater.cc:347] Connected to a master server at 127.9.225.62:43621
I20260809 00:52:08.868618 11646 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:08.869576 11646 heartbeater.cc:511] Master 127.9.225.62:43621 requested a full tablet report, sending...
I20260809 00:52:08.871170 11081 ts_manager.cc:194] Registered new tserver with Master: e70c879491044bd08444023659b64479 (127.9.225.2:37557)
I20260809 00:52:08.872138 11081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:53745
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:52:08.906562 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43621
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:09.296324 11658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:09.296813 11658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:09.297109 11658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:09.307402 11658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:09.307559 11658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:09.307857 11658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:52:09.323206 11658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43621
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:09.324685 11658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:09.326300 11658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:09.345031 11668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:09.345165 11669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:09.345206 11671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:09.345875 11414 heartbeater.cc:503] Master 127.9.225.62:43621 was elected leader, sending a full tablet report...
I20260809 00:52:09.351432 11658 server_base.cc:1034] running on GCE node
I20260809 00:52:09.352286 11658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:09.353674 11658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:09.354975 11658 hybrid_clock.cc:648] HybridClock initialized: now 1786236729354856 us; error 66 us; skew 500 ppm
I20260809 00:52:09.355413 11658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:09 dist-test-slave-j2vl krb5kdc[11005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236729, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:09.360123 11658 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:52:09.362470 11658 webserver.cc:460] Webserver started at http://127.9.225.3:45343/ using document root <none> and password file <none>
I20260809 00:52:09.363541 11658 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:09.363879 11658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:09.364390 11658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:09.368420 11658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "7fbb7fe86d6d45d3b5805dad0349a64d"
format_stamp: "Formatted at 2026-08-09 00:52:09 on dist-test-slave-j2vl"
server_key: "aa7ef625db8345a08bed704f478a70fb"
server_key_iv: "c4f1dcd1839e113922d59bdfe58433ac"
server_key_version: "encryptionkey@0"
I20260809 00:52:09.369550 11658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "7fbb7fe86d6d45d3b5805dad0349a64d"
format_stamp: "Formatted at 2026-08-09 00:52:09 on dist-test-slave-j2vl"
server_key: "aa7ef625db8345a08bed704f478a70fb"
server_key_iv: "c4f1dcd1839e113922d59bdfe58433ac"
server_key_version: "encryptionkey@0"
I20260809 00:52:09.376358 11658 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260809 00:52:09.381752 11680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:09.383335 11658 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260809 00:52:09.383561 11658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "7fbb7fe86d6d45d3b5805dad0349a64d"
format_stamp: "Formatted at 2026-08-09 00:52:09 on dist-test-slave-j2vl"
server_key: "aa7ef625db8345a08bed704f478a70fb"
server_key_iv: "c4f1dcd1839e113922d59bdfe58433ac"
server_key_version: "encryptionkey@0"
I20260809 00:52:09.383807 11658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:09.407721 11658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:09.413810 11658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:09.414263 11658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:09.415699 11658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:09.417991 11658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:09.418170 11658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:09.418376 11658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:09.418478 11658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:09.494432 11658 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:43051
I20260809 00:52:09.494472 11869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:43051 every 8 connection(s)
I20260809 00:52:09.496241 11658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:52:09.502389 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 11658
I20260809 00:52:09.502701 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:52:09.503216 10116 external_mini_cluster.cc:1428] Setting key 8054dc0ff1a96f8aa1c75a656da05ad1
Aug 09 00:52:09 dist-test-slave-j2vl krb5kdc[11005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236729, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:09.524394 11874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.500134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:40589 (local address 127.9.225.62:43621)
0809 00:52:09.500866 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:09.500870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:09.502371 (+ 1501us) server_negotiation.cc:407] Connection header received
0809 00:52:09.505461 (+ 3090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:09.505466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:09.505606 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:09.505731 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:09.508002 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.509009 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.510918 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.511518 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.511622 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.516063 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:09.516069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:09.516073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:09.516115 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:09.520706 (+ 4591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:09.521719 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:09.521724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:09.521726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:09.521920 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:09.522412 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:09.522416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:09.522418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:09.522759 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:09.522948 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:09.523751 (+ 803us) server_negotiation.cc:299] Negotiation successful
0809 00:52:09.523937 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":426,"threads_started":1}
I20260809 00:52:09.525579 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.500853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43621 (local address 127.9.225.3:40589)
0809 00:52:09.502199 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:09.502241 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:09.505089 (+ 2848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:09.506117 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:09.506131 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:09.507071 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:09.507794 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:09.507816 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.509187 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.509191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:09.510790 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:09.510804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.511756 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.511762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:09.512009 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.512791 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:09.512823 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:09.515743 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:09.520881 (+ 5138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:09.520890 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:09.520923 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:09.521583 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:09.522025 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:09.522028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:09.522030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:09.522268 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:09.522924 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:09.522931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:09.523315 (+ 384us) client_negotiation.cc:769] Sending connection context
0809 00:52:09.523582 (+ 267us) client_negotiation.cc:240] Negotiation successful
0809 00:52:09.524059 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"thread_start_us":709,"threads_started":1}
I20260809 00:52:09.527273 11872 heartbeater.cc:347] Connected to a master server at 127.9.225.62:43621
I20260809 00:52:09.527659 11872 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:09.528707 11872 heartbeater.cc:511] Master 127.9.225.62:43621 requested a full tablet report, sending...
I20260809 00:52:09.530709 11081 ts_manager.cc:194] Registered new tserver with Master: 7fbb7fe86d6d45d3b5805dad0349a64d (127.9.225.3:43051)
I20260809 00:52:09.531855 11081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:40589
I20260809 00:52:09.533288 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:52:09.559672 11874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.541875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.9.225.62:43621)
0809 00:52:09.542072 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:09.542077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:09.543107 (+ 1030us) server_negotiation.cc:407] Connection header received
0809 00:52:09.543398 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:09.543402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:09.543479 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:09.543590 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:09.546247 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.547157 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.549002 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.549707 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.549846 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.553376 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:09.553385 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:09.553390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:09.553473 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:09.556669 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:09.557436 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:09.557441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:09.557448 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:09.557617 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:09.558015 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:09.558020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:09.558024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:09.558553 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:09.558765 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:09.559157 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 00:52:09.559354 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 00:52:09.569831 11081 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:52:09.581894 11081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:52:09.631500 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.602768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:37557 (local address 127.0.0.1:42996)
0809 00:52:09.618792 (+ 16024us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:09.618823 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:09.619144 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:09.620096 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:09.620104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:09.620134 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:09.621145 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:09.621170 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.624245 (+ 3075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.624251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:09.627026 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:09.627040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.629869 (+ 2829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.629879 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:09.630083 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.630144 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 00:52:09.630278 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 00:52:09.630561 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15850,"mutex_wait_us":48,"thread_start_us":15589,"threads_started":1}
I20260809 00:52:09.631889 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.603544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.9.225.2:37557)
0809 00:52:09.612081 (+ 8537us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:09.612095 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:09.618955 (+ 6860us) server_negotiation.cc:407] Connection header received
0809 00:52:09.619380 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:09.619391 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:09.619772 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:09.620081 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:09.621426 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.624067 (+ 2641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.627354 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.629593 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.629779 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.630058 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:09.631219 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0809 00:52:09.631484 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8263,"thread_start_us":8111,"threads_started":1}
I20260809 00:52:09.631999 11899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.602281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.9.225.3:43051)
0809 00:52:09.619642 (+ 17361us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:09.619652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:09.619696 (+ 44us) server_negotiation.cc:407] Connection header received
0809 00:52:09.619812 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:09.619820 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:09.620063 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:09.620736 (+ 673us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:09.622312 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.624771 (+ 2459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.628454 (+ 3683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.629974 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.630099 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.630420 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:09.630878 (+ 458us) server_negotiation.cc:299] Negotiation successful
0809 00:52:09.631117 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17138,"thread_start_us":16904,"threads_started":1}
I20260809 00:52:09.632345 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.601977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:43051 (local address 127.0.0.1:56740)
0809 00:52:09.614747 (+ 12770us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:09.614875 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:09.615254 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:09.620301 (+ 5047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:09.620305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:09.620331 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:09.621906 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:09.621938 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.624940 (+ 3002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.624945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:09.628189 (+ 3244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:09.628205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.630328 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.630337 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:09.630631 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.630675 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 00:52:09.630821 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 00:52:09.630988 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10970,"mutex_wait_us":5,"thread_start_us":10584,"threads_started":1}
I20260809 00:52:09.633092 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.602183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:33891 (local address 127.0.0.1:53152)
0809 00:52:09.618378 (+ 16195us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:09.618405 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:09.619622 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:09.620549 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:09.620552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:09.620576 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:09.621971 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:09.621982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.624429 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.624437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:09.628606 (+ 4169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:09.628620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.631881 (+ 3261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.631890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:09.632163 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.632196 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 00:52:09.632300 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 00:52:09.632465 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15905,"mutex_wait_us":123,"spinlock_wait_cycles":42368,"thread_start_us":15446,"threads_started":1}
I20260809 00:52:09.634109 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.602835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.9.225.1:33891)
0809 00:52:09.616927 (+ 14092us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:09.616942 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:09.618569 (+ 1627us) server_negotiation.cc:407] Connection header received
0809 00:52:09.619875 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:09.619883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:09.620062 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:09.620283 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:09.622240 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.624224 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.628830 (+ 4606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.631033 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.631248 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.631619 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:09.633470 (+ 1851us) server_negotiation.cc:299] Negotiation successful
0809 00:52:09.633681 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13738,"thread_start_us":13995,"threads_started":1}
I20260809 00:52:09.637909 11537 tablet_service.cc:1467] Processing CreateTablet for tablet fe14dba2d7eb4fad94b2e30946cb7bdd (DEFAULT_TABLE table=test-table [id=96a582d40c634348951d376d2441caaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:09.639169 11294 tablet_service.cc:1467] Processing CreateTablet for tablet fe14dba2d7eb4fad94b2e30946cb7bdd (DEFAULT_TABLE table=test-table [id=96a582d40c634348951d376d2441caaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:09.639550 11759 tablet_service.cc:1467] Processing CreateTablet for tablet fe14dba2d7eb4fad94b2e30946cb7bdd (DEFAULT_TABLE table=test-table [id=96a582d40c634348951d376d2441caaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:09.640514 11537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe14dba2d7eb4fad94b2e30946cb7bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:09.640990 11294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe14dba2d7eb4fad94b2e30946cb7bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:09.641608 11759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe14dba2d7eb4fad94b2e30946cb7bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:09.667016 11908 tablet_bootstrap.cc:492] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d: Bootstrap starting.
I20260809 00:52:09.667011 11911 tablet_bootstrap.cc:492] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479: Bootstrap starting.
I20260809 00:52:09.670419 11909 tablet_bootstrap.cc:492] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f: Bootstrap starting.
I20260809 00:52:09.674431 11911 tablet_bootstrap.cc:654] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:09.674470 11908 tablet_bootstrap.cc:654] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:09.676517 11908 log.cc:824] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:09.676537 11911 log.cc:824] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:09.676597 11909 tablet_bootstrap.cc:654] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:09.678543 11909 log.cc:824] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:09.681747 11911 tablet_bootstrap.cc:492] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479: No bootstrap required, opened a new log
I20260809 00:52:09.682173 11911 ts_tablet_manager.cc:1397] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260809 00:52:09.682539 11909 tablet_bootstrap.cc:492] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f: No bootstrap required, opened a new log
I20260809 00:52:09.682868 11908 tablet_bootstrap.cc:492] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d: No bootstrap required, opened a new log
I20260809 00:52:09.682893 11909 ts_tablet_manager.cc:1397] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260809 00:52:09.683270 11908 ts_tablet_manager.cc:1397] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.006s
I20260809 00:52:09.693913 11909 raft_consensus.cc:348] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbb7fe86d6d45d3b5805dad0349a64d" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43051 } } peers { permanent_uuid: "9015ad79e31c4deb8f70c6ad695db55f" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33891 } } peers { permanent_uuid: "e70c879491044bd08444023659b64479" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37557 } }
I20260809 00:52:09.694514 11909 raft_consensus.cc:374] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:09.694748 11909 raft_consensus.cc:729] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9015ad79e31c4deb8f70c6ad695db55f, State: Initialized, Role: FOLLOWER
I20260809 00:52:09.695699 11909 consensus_queue.cc:260] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbb7fe86d6d45d3b5805dad0349a64d" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43051 } } peers { permanent_uuid: "9015ad79e31c4deb8f70c6ad695db55f" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33891 } } peers { permanent_uuid: "e70c879491044bd08444023659b64479" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37557 } }
I20260809 00:52:09.697705 11911 raft_consensus.cc:348] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbb7fe86d6d45d3b5805dad0349a64d" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43051 } } peers { permanent_uuid: "9015ad79e31c4deb8f70c6ad695db55f" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33891 } } peers { permanent_uuid: "e70c879491044bd08444023659b64479" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37557 } }
I20260809 00:52:09.698567 11911 raft_consensus.cc:374] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:09.698817 11911 raft_consensus.cc:729] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70c879491044bd08444023659b64479, State: Initialized, Role: FOLLOWER
I20260809 00:52:09.699258 11908 raft_consensus.cc:348] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbb7fe86d6d45d3b5805dad0349a64d" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43051 } } peers { permanent_uuid: "9015ad79e31c4deb8f70c6ad695db55f" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33891 } } peers { permanent_uuid: "e70c879491044bd08444023659b64479" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37557 } }
I20260809 00:52:09.699841 11908 raft_consensus.cc:374] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:09.700048 11908 raft_consensus.cc:729] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbb7fe86d6d45d3b5805dad0349a64d, State: Initialized, Role: FOLLOWER
I20260809 00:52:09.700060 11911 consensus_queue.cc:260] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbb7fe86d6d45d3b5805dad0349a64d" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43051 } } peers { permanent_uuid: "9015ad79e31c4deb8f70c6ad695db55f" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33891 } } peers { permanent_uuid: "e70c879491044bd08444023659b64479" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37557 } }
I20260809 00:52:09.700560 11909 ts_tablet_manager.cc:1428] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
I20260809 00:52:09.701061 11908 consensus_queue.cc:260] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbb7fe86d6d45d3b5805dad0349a64d" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43051 } } peers { permanent_uuid: "9015ad79e31c4deb8f70c6ad695db55f" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33891 } } peers { permanent_uuid: "e70c879491044bd08444023659b64479" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37557 } }
I20260809 00:52:09.704532 11872 heartbeater.cc:503] Master 127.9.225.62:43621 was elected leader, sending a full tablet report...
I20260809 00:52:09.705945 11908 ts_tablet_manager.cc:1428] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d: Time spent starting tablet: real 0.022s user 0.012s sys 0.011s
I20260809 00:52:09.707954 11646 heartbeater.cc:503] Master 127.9.225.62:43621 was elected leader, sending a full tablet report...
I20260809 00:52:09.709105 11911 ts_tablet_manager.cc:1428] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479: Time spent starting tablet: real 0.027s user 0.014s sys 0.011s
W20260809 00:52:09.749186 11873 tablet.cc:2369] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:52:09.812281 11415 tablet.cc:2369] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:52:09.842607 11647 tablet.cc:2369] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:52:09.948571 11919 raft_consensus.cc:482] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:52:09.949092 11919 raft_consensus.cc:504] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbb7fe86d6d45d3b5805dad0349a64d" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43051 } } peers { permanent_uuid: "9015ad79e31c4deb8f70c6ad695db55f" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33891 } } peers { permanent_uuid: "e70c879491044bd08444023659b64479" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37557 } }
I20260809 00:52:09.952051 11919 leader_election.cc:288] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fbb7fe86d6d45d3b5805dad0349a64d (127.9.225.3:43051), e70c879491044bd08444023659b64479 (127.9.225.2:37557)
I20260809 00:52:09.962337 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.953545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:48459 (local address 127.9.225.2:37557)
0809 00:52:09.953852 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:09.953860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:09.954870 (+ 1010us) server_negotiation.cc:407] Connection header received
0809 00:52:09.955158 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:09.955164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:09.955236 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:09.955335 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:09.956389 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.957242 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.959914 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.961053 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.961244 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.961381 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:09.961755 (+ 374us) server_negotiation.cc:299] Negotiation successful
0809 00:52:09.961893 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 00:52:09.962774 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.952264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:37557 (local address 127.9.225.1:48459)
0809 00:52:09.954731 (+ 2467us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:09.954753 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:09.954990 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:09.955353 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:09.955358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:09.955524 (+ 166us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:09.956234 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:09.956246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.957404 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.957412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:09.959723 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:09.959740 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.961369 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.961376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:09.961577 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.961602 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 00:52:09.961721 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 00:52:09.961867 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2300,"mutex_wait_us":1,"spinlock_wait_cycles":104704,"thread_start_us":1666,"threads_started":1}
I20260809 00:52:09.963231 11899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.952264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:35113 (local address 127.9.225.3:43051)
0809 00:52:09.952499 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:09.952505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:09.955357 (+ 2852us) server_negotiation.cc:407] Connection header received
0809 00:52:09.955605 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:09.955609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:09.955674 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:09.955776 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:09.956889 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.957903 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.960800 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.962004 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.962168 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.962311 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:09.962768 (+ 457us) server_negotiation.cc:299] Negotiation successful
0809 00:52:09.962911 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 00:52:09.963322 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.952313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:43051 (local address 127.9.225.1:35113)
0809 00:52:09.955190 (+ 2877us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:09.955231 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:09.955433 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:09.955794 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:09.955798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:09.955829 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:09.956705 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:09.956717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.958113 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.958117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:09.960535 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:09.960549 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.962349 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.962362 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:09.962604 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:09.962634 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 00:52:09.962738 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 00:52:09.962872 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2749,"thread_start_us":2323,"threads_started":1}
I20260809 00:52:09.964600 11802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe14dba2d7eb4fad94b2e30946cb7bdd" candidate_uuid: "9015ad79e31c4deb8f70c6ad695db55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbb7fe86d6d45d3b5805dad0349a64d" is_pre_election: true
I20260809 00:52:09.965296 11802 raft_consensus.cc:2454] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9015ad79e31c4deb8f70c6ad695db55f in term 0.
I20260809 00:52:09.966121 11221 leader_election.cc:302] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fbb7fe86d6d45d3b5805dad0349a64d, 9015ad79e31c4deb8f70c6ad695db55f; no voters:
I20260809 00:52:09.966626 11919 raft_consensus.cc:2781] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:52:09.966804 11919 raft_consensus.cc:482] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:52:09.966634 11573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe14dba2d7eb4fad94b2e30946cb7bdd" candidate_uuid: "9015ad79e31c4deb8f70c6ad695db55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e70c879491044bd08444023659b64479" is_pre_election: true
I20260809 00:52:09.966984 11919 raft_consensus.cc:3037] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:09.967343 11573 raft_consensus.cc:2454] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9015ad79e31c4deb8f70c6ad695db55f in term 0.
I20260809 00:52:09.969838 11919 raft_consensus.cc:504] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbb7fe86d6d45d3b5805dad0349a64d" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43051 } } peers { permanent_uuid: "9015ad79e31c4deb8f70c6ad695db55f" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33891 } } peers { permanent_uuid: "e70c879491044bd08444023659b64479" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37557 } }
I20260809 00:52:09.970733 11919 leader_election.cc:288] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [CANDIDATE]: Term 1 election: Requested vote from peers 7fbb7fe86d6d45d3b5805dad0349a64d (127.9.225.3:43051), e70c879491044bd08444023659b64479 (127.9.225.2:37557)
I20260809 00:52:09.971212 11802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe14dba2d7eb4fad94b2e30946cb7bdd" candidate_uuid: "9015ad79e31c4deb8f70c6ad695db55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbb7fe86d6d45d3b5805dad0349a64d"
I20260809 00:52:09.971298 11573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe14dba2d7eb4fad94b2e30946cb7bdd" candidate_uuid: "9015ad79e31c4deb8f70c6ad695db55f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e70c879491044bd08444023659b64479"
I20260809 00:52:09.971484 11802 raft_consensus.cc:3037] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:09.971576 11573 raft_consensus.cc:3037] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:09.974306 11802 raft_consensus.cc:2454] T fe14dba2d7eb4fad94b2e30946cb7bdd P 7fbb7fe86d6d45d3b5805dad0349a64d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9015ad79e31c4deb8f70c6ad695db55f in term 1.
I20260809 00:52:09.974323 11573 raft_consensus.cc:2454] T fe14dba2d7eb4fad94b2e30946cb7bdd P e70c879491044bd08444023659b64479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9015ad79e31c4deb8f70c6ad695db55f in term 1.
I20260809 00:52:09.974823 11221 leader_election.cc:302] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7fbb7fe86d6d45d3b5805dad0349a64d, 9015ad79e31c4deb8f70c6ad695db55f, e70c879491044bd08444023659b64479; no voters:
I20260809 00:52:09.975248 11919 raft_consensus.cc:2781] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:09.975833 11919 raft_consensus.cc:686] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [term 1 LEADER]: Becoming Leader. State: Replica: 9015ad79e31c4deb8f70c6ad695db55f, State: Running, Role: LEADER
I20260809 00:52:09.976449 11919 consensus_queue.cc:237] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbb7fe86d6d45d3b5805dad0349a64d" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43051 } } peers { permanent_uuid: "9015ad79e31c4deb8f70c6ad695db55f" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33891 } } peers { permanent_uuid: "e70c879491044bd08444023659b64479" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37557 } }
I20260809 00:52:09.982465 11077 catalog_manager.cc:5495] T fe14dba2d7eb4fad94b2e30946cb7bdd P 9015ad79e31c4deb8f70c6ad695db55f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9015ad79e31c4deb8f70c6ad695db55f (127.9.225.1). New cstate: current_term: 1 leader_uuid: "9015ad79e31c4deb8f70c6ad695db55f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbb7fe86d6d45d3b5805dad0349a64d" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9015ad79e31c4deb8f70c6ad695db55f" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e70c879491044bd08444023659b64479" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37557 } health_report { overall_health: UNKNOWN } } }
Aug 09 00:52:10 dist-test-slave-j2vl krb5kdc[11005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:10.018700 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:09.992772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.9.225.1:33891)
0809 00:52:09.993022 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:09.993027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:09.994055 (+ 1028us) server_negotiation.cc:407] Connection header received
0809 00:52:09.994291 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:09.994295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:09.994363 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:09.994476 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:09.996882 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:09.997990 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:09.999877 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:10.000726 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:10.000906 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:10.011188 (+ 10282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:10.011209 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:10.011229 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:10.011275 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:10.015073 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:10.016000 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:10.016009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:10.016012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:10.016232 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:10.016692 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:10.016704 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:10.016707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:10.017127 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:10.017550 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:10.017837 (+ 287us) server_negotiation.cc:299] Negotiation successful
0809 00:52:10.018187 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 00:52:10 dist-test-slave-j2vl krb5kdc[11005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236730, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 00:52:10 dist-test-slave-j2vl krb5kdc[11005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236730, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:10.079250 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:10.062706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.9.225.1:33891)
0809 00:52:10.062963 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:10.062973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:10.063407 (+ 434us) server_negotiation.cc:407] Connection header received
0809 00:52:10.063580 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:10.063584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:10.063653 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:10.063756 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:10.065302 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:10.066317 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:10.067593 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:10.068277 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:10.068386 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:10.072974 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:10.072980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:10.072983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:10.073025 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:10.076139 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:10.076886 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:10.076892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:10.076897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:10.077083 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:10.077608 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:10.077614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:10.077619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:10.077936 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:10.078281 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:10.078576 (+ 295us) server_negotiation.cc:299] Negotiation successful
0809 00:52:10.078838 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 00:52:10 dist-test-slave-j2vl krb5kdc[11005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 00:52:10 dist-test-slave-j2vl krb5kdc[11005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:10.137652 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:10.120046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.9.225.1:33891)
0809 00:52:10.120264 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:10.120272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:10.120720 (+ 448us) server_negotiation.cc:407] Connection header received
0809 00:52:10.120952 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:10.120956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:10.121045 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:10.121159 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:10.122966 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:10.123770 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:10.125368 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:10.125939 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:10.126096 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:10.131053 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:10.131059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:10.131063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:10.131103 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:10.134380 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:10.135091 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:10.135096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:10.135106 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:10.135294 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:10.135813 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:10.135819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:10.135822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:10.136180 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:10.136588 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:10.136866 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 00:52:10.137185 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260809 00:52:10.140125 11262 tablet_service.cc:2908] Rejecting scan request for tablet fe14dba2d7eb4fad94b2e30946cb7bdd: Uninitialized: clean time has not yet been initialized
I20260809 00:52:10.144343 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11188
I20260809 00:52:10.161109 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:10.145713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53184 (local address 127.9.225.1:33891)
0809 00:52:10.145931 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:10.145937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:10.146147 (+ 210us) server_negotiation.cc:407] Connection header received
0809 00:52:10.146273 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:10.146277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:10.146349 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:10.146440 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:10.148118 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:10.148963 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:10.150459 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:10.151116 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:10.151222 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:10.154112 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:10.154117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:10.154120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:10.154157 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:10.157884 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:10.158749 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:10.158753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:10.158756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:10.158939 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:10.159331 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:10.159334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:10.159336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:10.159731 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:10.160048 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:10.160289 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 00:52:10.160629 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 00:52:10.161903 11405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:10.259176 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 11188
I20260809 00:52:10.290871 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11427
Aug 09 00:52:10 dist-test-slave-j2vl krb5kdc[11005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:52:10.306391 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:10.291690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.9.225.2:37557)
0809 00:52:10.291912 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:10.291917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:10.291937 (+ 20us) server_negotiation.cc:407] Connection header received
0809 00:52:10.292070 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:10.292074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:10.292142 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:10.292230 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:10.293592 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:10.294360 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:10.295752 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:10.296170 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:10.296350 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:10.300949 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:10.300967 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:10.300988 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:10.301036 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:10.303856 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:10.304511 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:10.304519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:10.304521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:10.304658 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:10.304989 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:10.304992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:10.304994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:10.305448 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:10.305679 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:10.305925 (+ 246us) server_negotiation.cc:299] Negotiation successful
0809 00:52:10.306102 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 00:52:10.307132 11636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:10.398896 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 11427
I20260809 00:52:10.425945 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11658
Aug 09 00:52:10 dist-test-slave-j2vl krb5kdc[11005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:52:10.441509 11899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:10.426790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.9.225.3:43051)
0809 00:52:10.427007 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:10.427014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:10.427037 (+ 23us) server_negotiation.cc:407] Connection header received
0809 00:52:10.427223 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:10.427229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:10.427328 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:10.427450 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:10.428712 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:10.429526 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:10.430793 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:10.431201 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:10.431335 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:10.435635 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:10.435652 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:10.435671 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:10.435708 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:10.438486 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:10.439210 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:10.439220 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:10.439222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:10.439394 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:10.439806 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:10.439812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:10.439815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:10.440195 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:10.440522 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:10.440815 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 00:52:10.441055 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 00:52:10.442442 11862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:10.536295 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 11658
I20260809 00:52:10.563706 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11029
I20260809 00:52:10.565018 11132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:10.679852 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 11029
2026-08-09T00:52:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5085 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:52:10 dist-test-slave-j2vl krb5kdc[11968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:52:10 dist-test-slave-j2vl krb5kdc[11968](info): set up 2 sockets
Aug 09 00:52:10 dist-test-slave-j2vl krb5kdc[11968](info): commencing operation
krb5kdc: starting...
W20260809 00:52:12.016276 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:52:12 dist-test-slave-j2vl krb5kdc[11968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:52:12.244762 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:38157
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:38157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:12.592453 11992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:12.592793 11992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:12.592890 11992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:12.600795 11992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:12.600934 11992 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 00:52:12.600978 11992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:12.601042 11992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:12.601092 11992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:12.601150 11992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:52:12.611744 11992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:38157
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:38157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:12.613103 11992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:12.614500 11992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:12.627059 12004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:12.627084 12005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:12.627334 12010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:12.628190 11992 server_base.cc:1034] running on GCE node
I20260809 00:52:12.629132 11992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:12.630848 11992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:12.632119 11992 hybrid_clock.cc:648] HybridClock initialized: now 1786236732632055 us; error 38 us; skew 500 ppm
I20260809 00:52:12.632498 11992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:12 dist-test-slave-j2vl krb5kdc[11968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236732, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:12.635883 11992 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:52:12.637501 11992 webserver.cc:460] Webserver started at http://127.9.225.62:34153/ using document root <none> and password file <none>
I20260809 00:52:12.638258 11992 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:12.638352 11992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:12.638661 11992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:12.641870 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "566efdc6fe83418896302d5d26098e50"
format_stamp: "Formatted at 2026-08-09 00:52:12 on dist-test-slave-j2vl"
server_key: "8343c26946d1be18cac8188981328b37"
server_key_iv: "b97a0e6cffb9fe069e8effa9847c9fa5"
server_key_version: "encryptionkey@0"
I20260809 00:52:12.642691 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "566efdc6fe83418896302d5d26098e50"
format_stamp: "Formatted at 2026-08-09 00:52:12 on dist-test-slave-j2vl"
server_key: "8343c26946d1be18cac8188981328b37"
server_key_iv: "b97a0e6cffb9fe069e8effa9847c9fa5"
server_key_version: "encryptionkey@0"
I20260809 00:52:12.648365 11992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 00:52:12.652173 12020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:12.653684 11992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 00:52:12.653841 11992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "566efdc6fe83418896302d5d26098e50"
format_stamp: "Formatted at 2026-08-09 00:52:12 on dist-test-slave-j2vl"
server_key: "8343c26946d1be18cac8188981328b37"
server_key_iv: "b97a0e6cffb9fe069e8effa9847c9fa5"
server_key_version: "encryptionkey@0"
I20260809 00:52:12.654036 11992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:12.691910 11992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:12.696969 11992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:12.697228 11992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:12.721133 11992 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:38157
I20260809 00:52:12.721141 12118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:38157 every 8 connection(s)
I20260809 00:52:12.722692 11992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:12.726886 12120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:12.727826 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 11992
I20260809 00:52:12.728112 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:12.728969 10116 external_mini_cluster.cc:1428] Setting key a969e8436cfb9432e0e232a3ab18a11d
I20260809 00:52:12.743256 12120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50: Bootstrap starting.
Aug 09 00:52:12 dist-test-slave-j2vl krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:12.747351 12120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:12.748768 12120 log.cc:824] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:12.751989 12120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50: No bootstrap required, opened a new log
I20260809 00:52:12.754819 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:12.731482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.9.225.62:38157)
0809 00:52:12.732467 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:12.732480 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:12.732530 (+ 50us) server_negotiation.cc:407] Connection header received
0809 00:52:12.734442 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:12.734477 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:12.734999 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:12.735451 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:12.737545 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:12.738913 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:12.740806 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:12.741606 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:12.741745 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:12.746796 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:12.746806 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:12.746823 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:12.746867 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:12.750288 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:12.751085 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:12.751092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:12.751099 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:12.751309 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:12.751732 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:12.751737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:12.751747 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:12.752378 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:12.752711 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:12.753182 (+ 471us) server_negotiation.cc:299] Negotiation successful
0809 00:52:12.753693 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":469,"threads_started":1}
I20260809 00:52:12.760632 12120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566efdc6fe83418896302d5d26098e50" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 38157 } }
I20260809 00:52:12.761050 12120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:12.761165 12120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 566efdc6fe83418896302d5d26098e50, State: Initialized, Role: FOLLOWER
I20260809 00:52:12.761974 12120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566efdc6fe83418896302d5d26098e50" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 38157 } }
I20260809 00:52:12.762275 12120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:12.762439 12120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:12.762643 12120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:12.765029 12120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566efdc6fe83418896302d5d26098e50" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 38157 } }
I20260809 00:52:12.765638 12120 leader_election.cc:302] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 566efdc6fe83418896302d5d26098e50; no voters:
I20260809 00:52:12.766317 12120 leader_election.cc:288] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:12.766450 12126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:12.767699 12126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [term 1 LEADER]: Becoming Leader. State: Replica: 566efdc6fe83418896302d5d26098e50, State: Running, Role: LEADER
I20260809 00:52:12.768303 12126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566efdc6fe83418896302d5d26098e50" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 38157 } }
I20260809 00:52:12.768957 12120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:12.775579 12131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "566efdc6fe83418896302d5d26098e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566efdc6fe83418896302d5d26098e50" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 38157 } } }
I20260809 00:52:12.776749 12131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:12.776667 12133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 566efdc6fe83418896302d5d26098e50. Latest consensus state: current_term: 1 leader_uuid: "566efdc6fe83418896302d5d26098e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "566efdc6fe83418896302d5d26098e50" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 38157 } } }
I20260809 00:52:12.780539 12133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:12.786355 12143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:12.792495 12143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:12.802809 12143 catalog_manager.cc:1347] Generated new cluster ID: ad4ab226c0d243328de0f753b05901e4
I20260809 00:52:12.802969 12143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:12.816453 12143 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:12.818024 12143 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:12.829058 12143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50: Generated new TSK 0
I20260809 00:52:12.830108 12143 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:52:12.890362 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:38157
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:13.246248 12169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:13.246580 12169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:13.246759 12169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:13.254742 12169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:13.254900 12169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:13.255066 12169 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 00:52:13.255199 12169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:52:13.266129 12169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.225.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:13.267583 12169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:13.269135 12169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:13.286768 12184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:13.288228 12183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:13.288679 12186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:13.289589 12169 server_base.cc:1034] running on GCE node
I20260809 00:52:13.290593 12169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:13.291775 12169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:13.293037 12169 hybrid_clock.cc:648] HybridClock initialized: now 1786236733292944 us; error 64 us; skew 500 ppm
I20260809 00:52:13.293551 12169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:13 dist-test-slave-j2vl krb5kdc[11968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236733, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:13.298198 12169 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:52:13.300297 12169 webserver.cc:460] Webserver started at http://127.9.225.1:38491/ using document root <none> and password file <none>
I20260809 00:52:13.301441 12169 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:13.301600 12169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:13.302042 12169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:13.307073 12169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "303026dfbcd044c0abe4bc0a11c92e24"
format_stamp: "Formatted at 2026-08-09 00:52:13 on dist-test-slave-j2vl"
server_key: "e649d42ec1a55ef37af0e0e91535aff5"
server_key_iv: "5369dc4d77890de87dd94803f384b074"
server_key_version: "encryptionkey@0"
I20260809 00:52:13.308656 12169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "303026dfbcd044c0abe4bc0a11c92e24"
format_stamp: "Formatted at 2026-08-09 00:52:13 on dist-test-slave-j2vl"
server_key: "e649d42ec1a55ef37af0e0e91535aff5"
server_key_iv: "5369dc4d77890de87dd94803f384b074"
server_key_version: "encryptionkey@0"
I20260809 00:52:13.316730 12169 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260809 00:52:13.322353 12200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:13.324559 12169 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 00:52:13.324777 12169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "303026dfbcd044c0abe4bc0a11c92e24"
format_stamp: "Formatted at 2026-08-09 00:52:13 on dist-test-slave-j2vl"
server_key: "e649d42ec1a55ef37af0e0e91535aff5"
server_key_iv: "5369dc4d77890de87dd94803f384b074"
server_key_version: "encryptionkey@0"
I20260809 00:52:13.325107 12169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:13.340000 12169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:13.344285 12169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:13.344621 12169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:13.347611 12169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:13.347698 12169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:13.347899 12169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:13.347962 12169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 00:52:13 dist-test-slave-j2vl krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236733, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:13.374071 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:13.348641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:49715 (local address 127.9.225.62:38157)
0809 00:52:13.349154 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:13.349161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:13.350704 (+ 1543us) server_negotiation.cc:407] Connection header received
0809 00:52:13.354118 (+ 3414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:13.354123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:13.354199 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:13.354341 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:13.356938 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.358356 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.360695 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.361618 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.361792 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:13.367032 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:13.367038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:13.367042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:13.367080 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:13.370690 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:13.371538 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:13.371545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:13.371549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:13.371682 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:13.372161 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:13.372168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:13.372174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:13.372452 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:13.372605 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:13.373421 (+ 816us) server_negotiation.cc:299] Negotiation successful
0809 00:52:13.373676 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":242,"threads_started":1}
I20260809 00:52:13.375886 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:13.349492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:38157 (local address 127.9.225.1:49715)
0809 00:52:13.350485 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:13.350541 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:13.353692 (+ 3151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:13.354618 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:13.354627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:13.355601 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:13.356632 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:13.356672 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.358617 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.358625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:13.360501 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:13.360526 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.361940 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.361950 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:13.362265 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:13.363116 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:13.363141 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:13.366693 (+ 3552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:13.370853 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:13.370864 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:13.370891 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:13.371389 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:13.371813 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:13.371822 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:13.371824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:13.372037 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:13.372592 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:13.372600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:13.372983 (+ 383us) client_negotiation.cc:769] Sending connection context
0809 00:52:13.373278 (+ 295us) client_negotiation.cc:240] Negotiation successful
0809 00:52:13.373964 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":442,"threads_started":1}
I20260809 00:52:13.399247 12169 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:42655
I20260809 00:52:13.399266 12401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:42655 every 8 connection(s)
I20260809 00:52:13.400940 12169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:52:13.404623 12402 heartbeater.cc:347] Connected to a master server at 127.9.225.62:38157
I20260809 00:52:13.404901 12402 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:13.405956 12402 heartbeater.cc:511] Master 127.9.225.62:38157 requested a full tablet report, sending...
I20260809 00:52:13.405997 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 12169
I20260809 00:52:13.406234 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:52:13.406510 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:13.389432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:36175 (local address 127.9.225.62:38157)
0809 00:52:13.389633 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:13.389638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:13.390241 (+ 603us) server_negotiation.cc:407] Connection header received
0809 00:52:13.390472 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:13.390480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:13.390589 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:13.390707 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:13.392745 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.393768 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.396590 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.397064 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.397172 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:13.400030 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:13.400035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:13.400038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:13.400076 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:13.403172 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:13.404205 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:13.404209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:13.404211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:13.404369 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:13.404823 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:13.404828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:13.404831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:13.405241 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:13.405444 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:13.405873 (+ 429us) server_negotiation.cc:299] Negotiation successful
0809 00:52:13.406137 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 00:52:13.406634 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:13.389396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:38157 (local address 127.9.225.1:36175)
0809 00:52:13.390114 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:13.390132 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:13.390312 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:13.390793 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:13.390801 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:13.391724 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:13.392485 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:13.392513 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.393944 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.393949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:13.396375 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:13.396405 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.397284 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.397292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:13.397537 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:13.398378 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:13.398406 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:13.399821 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:13.403312 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:13.403316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:13.403319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:13.403873 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:13.404503 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:13.404508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:13.404512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:13.404673 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:13.405394 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:13.405399 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:13.405601 (+ 202us) client_negotiation.cc:769] Sending connection context
0809 00:52:13.405759 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 00:52:13.406137 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":3,"thread_start_us":414,"threads_started":1}
I20260809 00:52:13.407090 10116 external_mini_cluster.cc:1428] Setting key cc63fe04eb8f74d950dacac33f1f85df
I20260809 00:52:13.409210 12046 ts_manager.cc:194] Registered new tserver with Master: 303026dfbcd044c0abe4bc0a11c92e24 (127.9.225.1:42655)
I20260809 00:52:13.411106 12046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:49715
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:52:13.456794 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:38157
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:13.807389 12406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:13.807785 12406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:13.807898 12406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:13.815346 12406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:13.815472 12406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:13.815560 12406 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 00:52:13.815645 12406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:52:13.826155 12406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.225.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:13.827924 12406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:13.829576 12406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:13.844183 12413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:13.844110 12414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:13.844110 12417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:13.845228 12406 server_base.cc:1034] running on GCE node
I20260809 00:52:13.846011 12406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:13.847098 12406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:13.848407 12406 hybrid_clock.cc:648] HybridClock initialized: now 1786236733848277 us; error 74 us; skew 500 ppm
I20260809 00:52:13.848752 12406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:13 dist-test-slave-j2vl krb5kdc[11968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236733, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:13.852159 12406 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:52:13.853852 12406 webserver.cc:460] Webserver started at http://127.9.225.2:46011/ using document root <none> and password file <none>
I20260809 00:52:13.854653 12406 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:13.854794 12406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:13.855165 12406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:13.858603 12406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "329d3b81bb7342f6a40618d09ccf8999"
format_stamp: "Formatted at 2026-08-09 00:52:13 on dist-test-slave-j2vl"
server_key: "8c49fb646d8e17605911c2ba412e3f48"
server_key_iv: "7019b0b6c224a99c78cf03eecdbaaee8"
server_key_version: "encryptionkey@0"
I20260809 00:52:13.859603 12406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "329d3b81bb7342f6a40618d09ccf8999"
format_stamp: "Formatted at 2026-08-09 00:52:13 on dist-test-slave-j2vl"
server_key: "8c49fb646d8e17605911c2ba412e3f48"
server_key_iv: "7019b0b6c224a99c78cf03eecdbaaee8"
server_key_version: "encryptionkey@0"
I20260809 00:52:13.865283 12406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 00:52:13.869313 12431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:13.871007 12406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 00:52:13.871208 12406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "329d3b81bb7342f6a40618d09ccf8999"
format_stamp: "Formatted at 2026-08-09 00:52:13 on dist-test-slave-j2vl"
server_key: "8c49fb646d8e17605911c2ba412e3f48"
server_key_iv: "7019b0b6c224a99c78cf03eecdbaaee8"
server_key_version: "encryptionkey@0"
I20260809 00:52:13.871450 12406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:13.905618 12406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:13.910138 12406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:13.910435 12406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:13.913439 12406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:13.913558 12406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:13.913735 12406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:13.913812 12406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 00:52:13 dist-test-slave-j2vl krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236733, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:13.940963 12447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:13.914260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:53175 (local address 127.9.225.62:38157)
0809 00:52:13.914817 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:13.914824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:13.916187 (+ 1363us) server_negotiation.cc:407] Connection header received
0809 00:52:13.918746 (+ 2559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:13.918756 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:13.918878 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:13.919025 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:13.921704 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.922854 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.926436 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.927139 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.927305 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:13.932429 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:13.932435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:13.932438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:13.932477 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:13.937022 (+ 4545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:13.938328 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:13.938333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:13.938336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:13.938478 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:13.939018 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:13.939022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:13.939024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:13.939314 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:13.939495 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:13.940306 (+ 811us) server_negotiation.cc:299] Negotiation successful
0809 00:52:13.940560 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":282,"threads_started":1}
I20260809 00:52:13.942760 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:13.914992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:38157 (local address 127.9.225.2:53175)
0809 00:52:13.915962 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:13.916034 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:13.918280 (+ 2246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:13.919391 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:13.919420 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:13.920343 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:13.921461 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:13.921501 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.923119 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.923131 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:13.926190 (+ 3059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:13.926216 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.927579 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.927585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:13.927808 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:13.928667 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:13.928695 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:13.932132 (+ 3437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:13.937236 (+ 5104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:13.937245 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:13.937268 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:13.938147 (+ 879us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:13.938599 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:13.938606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:13.938610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:13.938874 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:13.939452 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:13.939462 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:13.939895 (+ 433us) client_negotiation.cc:769] Sending connection context
0809 00:52:13.940179 (+ 284us) client_negotiation.cc:240] Negotiation successful
0809 00:52:13.940778 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":429,"threads_started":1}
I20260809 00:52:13.967175 12447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:13.948848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:50021 (local address 127.9.225.62:38157)
0809 00:52:13.949092 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:13.949102 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:13.949828 (+ 726us) server_negotiation.cc:407] Connection header received
0809 00:52:13.950039 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:13.950046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:13.950145 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:13.950265 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:13.952061 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.953012 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.955927 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.956374 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.956490 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:13.959826 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:13.959832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:13.959835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:13.959873 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:13.963210 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:13.964278 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:13.964287 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:13.964290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:13.964495 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:13.965093 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:13.965099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:13.965102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:13.965695 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:13.965899 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:13.966724 (+ 825us) server_negotiation.cc:299] Negotiation successful
0809 00:52:13.966939 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 00:52:13.967765 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:13.948813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:38157 (local address 127.9.225.2:50021)
0809 00:52:13.949648 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:13.949677 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:13.949870 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:13.950300 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:13.950305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:13.951303 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:13.951890 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:13.951907 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.953199 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.953208 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:13.955690 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:13.955717 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:13.956706 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:13.956717 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:13.957023 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:13.957977 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:13.958041 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:13.959560 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:13.963386 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:13.963399 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:13.963406 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:13.964089 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:13.964661 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:13.964668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:13.964674 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:13.964906 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:13.965977 (+ 1071us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:13.965988 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:13.966327 (+ 339us) client_negotiation.cc:769] Sending connection context
0809 00:52:13.966547 (+ 220us) client_negotiation.cc:240] Negotiation successful
0809 00:52:13.967065 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":416,"threads_started":1}
I20260809 00:52:13.975041 12406 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:37587
I20260809 00:52:13.975081 12626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:37587 every 8 connection(s)
I20260809 00:52:13.976495 12406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:52:13.978835 12627 heartbeater.cc:347] Connected to a master server at 127.9.225.62:38157
I20260809 00:52:13.979152 12627 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:13.979863 12627 heartbeater.cc:511] Master 127.9.225.62:38157 requested a full tablet report, sending...
I20260809 00:52:13.981527 12046 ts_manager.cc:194] Registered new tserver with Master: 329d3b81bb7342f6a40618d09ccf8999 (127.9.225.2:37587)
I20260809 00:52:13.982143 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 12406
I20260809 00:52:13.982352 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:52:13.982574 12046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:53175
I20260809 00:52:13.982841 10116 external_mini_cluster.cc:1428] Setting key a663d14e47a43d4a733be8906b041562
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:52:14.033731 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:38157
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:14.392479 12634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:14.392863 12634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:14.393061 12634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:14.400962 12634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:14.401137 12634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:14.401296 12634 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 00:52:14.401465 12634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:52:14.411818 12634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.225.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:14.413403 12634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:14.414088 12402 heartbeater.cc:503] Master 127.9.225.62:38157 was elected leader, sending a full tablet report...
I20260809 00:52:14.415257 12634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:14.430140 12648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:14.430078 12643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:14.430174 12645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:14.431164 12634 server_base.cc:1034] running on GCE node
I20260809 00:52:14.431931 12634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:14.433063 12634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:14.434325 12634 hybrid_clock.cc:648] HybridClock initialized: now 1786236734434240 us; error 76 us; skew 500 ppm
I20260809 00:52:14.434677 12634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:14 dist-test-slave-j2vl krb5kdc[11968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236734, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:14.438216 12634 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:52:14.439917 12634 webserver.cc:460] Webserver started at http://127.9.225.3:40065/ using document root <none> and password file <none>
I20260809 00:52:14.440696 12634 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:14.440840 12634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:14.441200 12634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:14.444375 12634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "935d4267580747898906973bfbc9fd06"
format_stamp: "Formatted at 2026-08-09 00:52:14 on dist-test-slave-j2vl"
server_key: "a7103a3ba60de23468b8701592aa480c"
server_key_iv: "3dd6d9a69f7527592b6ed1e5360f7d48"
server_key_version: "encryptionkey@0"
I20260809 00:52:14.445271 12634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "935d4267580747898906973bfbc9fd06"
format_stamp: "Formatted at 2026-08-09 00:52:14 on dist-test-slave-j2vl"
server_key: "a7103a3ba60de23468b8701592aa480c"
server_key_iv: "3dd6d9a69f7527592b6ed1e5360f7d48"
server_key_version: "encryptionkey@0"
I20260809 00:52:14.450860 12634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 00:52:14.454680 12657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:14.456223 12634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 00:52:14.456377 12634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "935d4267580747898906973bfbc9fd06"
format_stamp: "Formatted at 2026-08-09 00:52:14 on dist-test-slave-j2vl"
server_key: "a7103a3ba60de23468b8701592aa480c"
server_key_iv: "3dd6d9a69f7527592b6ed1e5360f7d48"
server_key_version: "encryptionkey@0"
I20260809 00:52:14.456579 12634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:14.490131 12634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:14.495028 12634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:14.495289 12634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:14.498332 12634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:14.498428 12634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:14.498638 12634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:14.498724 12634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 00:52:14 dist-test-slave-j2vl krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236734, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:14.526269 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:14.498819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:41029 (local address 127.9.225.62:38157)
0809 00:52:14.499312 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:14.499318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:14.500641 (+ 1323us) server_negotiation.cc:407] Connection header received
0809 00:52:14.503013 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:14.503019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:14.503135 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:14.503261 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:14.505708 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.507135 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.509701 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.510327 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.510475 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:14.516452 (+ 5977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:14.516459 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:14.516462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:14.516501 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:14.521113 (+ 4612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:14.522321 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:14.522331 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:14.522337 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:14.522571 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:14.523294 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:14.523299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:14.523307 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:14.523751 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:14.523934 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:14.525217 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0809 00:52:14.525637 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":236,"threads_started":1}
I20260809 00:52:14.526959 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:14.499488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:38157 (local address 127.9.225.3:41029)
0809 00:52:14.500433 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:14.500486 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:14.502715 (+ 2229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:14.503562 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:14.503577 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:14.504467 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:14.505414 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:14.505454 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.507365 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.507373 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:14.509532 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:14.509547 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.510624 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.510635 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:14.511057 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:14.511951 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:14.511981 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:14.516103 (+ 4122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:14.521424 (+ 5321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:14.521442 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:14.521478 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:14.522144 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:14.522782 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:14.522790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:14.522796 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:14.523141 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:14.523899 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:14.523909 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:14.524570 (+ 661us) client_negotiation.cc:769] Sending connection context
0809 00:52:14.525043 (+ 473us) client_negotiation.cc:240] Negotiation successful
0809 00:52:14.525634 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":401,"threads_started":1}
I20260809 00:52:14.553304 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:14.536372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:52913 (local address 127.9.225.62:38157)
0809 00:52:14.536589 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:14.536599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:14.537399 (+ 800us) server_negotiation.cc:407] Connection header received
0809 00:52:14.537649 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:14.537657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:14.537742 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:14.537880 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:14.539551 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.540295 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.542673 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.543261 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.543411 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:14.546815 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:14.546820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:14.546823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:14.546859 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:14.549897 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:14.551012 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:14.551017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:14.551019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:14.551151 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:14.551678 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:14.551683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:14.551686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:14.552055 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:14.552228 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:14.552854 (+ 626us) server_negotiation.cc:299] Negotiation successful
0809 00:52:14.553049 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 00:52:14.554018 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:14.536395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:38157 (local address 127.9.225.3:52913)
0809 00:52:14.537237 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:14.537257 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:14.537496 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:14.537963 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:14.537972 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:14.538769 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:14.539326 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:14.539346 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.540471 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.540479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:14.542477 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:14.542496 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.543553 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.543568 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:14.543845 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:14.544830 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:14.544848 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:14.546566 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:14.550080 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:14.550088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:14.550095 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:14.550842 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:14.551284 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:14.551289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:14.551293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:14.551513 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:14.552213 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:14.552226 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:14.552531 (+ 305us) client_negotiation.cc:769] Sending connection context
0809 00:52:14.552734 (+ 203us) client_negotiation.cc:240] Negotiation successful
0809 00:52:14.553262 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":442,"threads_started":1}
I20260809 00:52:14.556836 12634 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:45767
I20260809 00:52:14.556854 12838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:45767 every 8 connection(s)
I20260809 00:52:14.558604 12634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:52:14.560868 12839 heartbeater.cc:347] Connected to a master server at 127.9.225.62:38157
I20260809 00:52:14.561131 12839 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:14.561892 12839 heartbeater.cc:511] Master 127.9.225.62:38157 requested a full tablet report, sending...
I20260809 00:52:14.563304 12046 ts_manager.cc:194] Registered new tserver with Master: 935d4267580747898906973bfbc9fd06 (127.9.225.3:45767)
I20260809 00:52:14.564167 12046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:41029
I20260809 00:52:14.568737 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 12634
I20260809 00:52:14.568913 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:52:14.569434 10116 external_mini_cluster.cc:1428] Setting key 8d3a10118c27c81e42925a3fb8806226
I20260809 00:52:14.572568 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 00:52:14 dist-test-slave-j2vl krb5kdc[11968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 00:52:14 dist-test-slave-j2vl krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:14.627990 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:14.613234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.9.225.62:38157)
0809 00:52:14.613512 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:14.613519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:14.613774 (+ 255us) server_negotiation.cc:407] Connection header received
0809 00:52:14.614016 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:14.614020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:14.614106 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:14.614227 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:14.615770 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.616494 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.617886 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.618332 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.618497 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:14.622635 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:14.622641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:14.622644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:14.622682 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:14.625547 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:14.626214 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:14.626219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:14.626221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:14.626348 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:14.626704 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:14.626708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:14.626710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:14.627001 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:14.627218 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:14.627452 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 00:52:14.627628 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 00:52:14.632377 12046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:52:14.639698 12046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:52:14.679384 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:14.653118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.9.225.2:37587)
0809 00:52:14.668578 (+ 15460us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:14.668589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:14.668627 (+ 38us) server_negotiation.cc:407] Connection header received
0809 00:52:14.669040 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:14.669050 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:14.669401 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:14.669661 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:14.670798 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.672541 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.675884 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.677093 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.677250 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:14.677516 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:14.678845 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0809 00:52:14.679054 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15225,"thread_start_us":15638,"threads_started":1}
I20260809 00:52:14.680145 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:14.652892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:37587 (local address 127.0.0.1:46642)
0809 00:52:14.663354 (+ 10462us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:14.663422 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:14.663640 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:14.669988 (+ 6348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:14.669998 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:14.670066 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:14.670551 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:14.670568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.672750 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.672756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:14.675680 (+ 2924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:14.675696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.677303 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.677310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:14.677869 (+ 559us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:14.677942 (+ 73us) client_negotiation.cc:769] Sending connection context
0809 00:52:14.678548 (+ 606us) client_negotiation.cc:240] Negotiation successful
0809 00:52:14.679691 (+ 1143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10167,"mutex_wait_us":1,"thread_start_us":21130,"threads_started":1}
I20260809 00:52:14.683580 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:14.655704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.9.225.1:42655)
0809 00:52:14.673309 (+ 17605us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:14.673318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:14.674635 (+ 1317us) server_negotiation.cc:407] Connection header received
0809 00:52:14.674773 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:14.674780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:14.675196 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:14.675385 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:14.676336 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.678012 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.681149 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.682570 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.682720 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:14.682913 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:14.683126 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 00:52:14.683293 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17426,"thread_start_us":17280,"threads_started":1}
I20260809 00:52:14.683584 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:14.655106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:42655 (local address 127.0.0.1:42748)
0809 00:52:14.674336 (+ 19230us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:14.674363 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:14.674507 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:14.675567 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:14.675574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:14.675610 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:14.676146 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:14.676160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.678169 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.678174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:14.680995 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:14.681008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.682776 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.682781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:14.682937 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:14.682976 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 00:52:14.683051 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 00:52:14.683154 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19087,"mutex_wait_us":6,"thread_start_us":18420,"threads_started":1}
I20260809 00:52:14.683812 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:14.653684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.9.225.3:45767)
0809 00:52:14.672575 (+ 18891us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:14.672587 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:14.674327 (+ 1740us) server_negotiation.cc:407] Connection header received
0809 00:52:14.674451 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:14.674460 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:14.674771 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:14.675003 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:14.676337 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.678040 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.680510 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.682115 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.682241 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:14.682507 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:14.683273 (+ 766us) server_negotiation.cc:299] Negotiation successful
0809 00:52:14.683451 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18669,"thread_start_us":18895,"threads_started":1}
I20260809 00:52:14.684393 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:14.653336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:45767 (local address 127.0.0.1:57842)
0809 00:52:14.673846 (+ 20510us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:14.674008 (+ 162us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:14.674170 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:14.675320 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:14.675325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:14.675350 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:14.676051 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:14.676066 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.678206 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.678210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:14.680272 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:14.680299 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:14.682662 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:14.682671 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:14.682860 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:14.682915 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 00:52:14.683011 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 00:52:14.683154 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20319,"mutex_wait_us":102,"thread_start_us":9531,"threads_started":1}
I20260809 00:52:14.684804 12518 tablet_service.cc:1467] Processing CreateTablet for tablet 4c5749f59f614fbc9b0fa60ad5af2bac (DEFAULT_TABLE table=test-table [id=8eef11e24b7640dd86f424394576c6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:14.686494 12518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c5749f59f614fbc9b0fa60ad5af2bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:14.687916 12736 tablet_service.cc:1467] Processing CreateTablet for tablet 4c5749f59f614fbc9b0fa60ad5af2bac (DEFAULT_TABLE table=test-table [id=8eef11e24b7640dd86f424394576c6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:14.687945 12284 tablet_service.cc:1467] Processing CreateTablet for tablet 4c5749f59f614fbc9b0fa60ad5af2bac (DEFAULT_TABLE table=test-table [id=8eef11e24b7640dd86f424394576c6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:14.689069 12736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c5749f59f614fbc9b0fa60ad5af2bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:14.689270 12284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c5749f59f614fbc9b0fa60ad5af2bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:14.701691 12872 tablet_bootstrap.cc:492] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06: Bootstrap starting.
I20260809 00:52:14.704967 12879 tablet_bootstrap.cc:492] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24: Bootstrap starting.
I20260809 00:52:14.707198 12872 tablet_bootstrap.cc:654] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:14.708037 12874 tablet_bootstrap.cc:492] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999: Bootstrap starting.
I20260809 00:52:14.708724 12872 log.cc:824] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:14.709414 12879 tablet_bootstrap.cc:654] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:14.711324 12879 log.cc:824] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:14.712754 12872 tablet_bootstrap.cc:492] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06: No bootstrap required, opened a new log
I20260809 00:52:14.713092 12872 ts_tablet_manager.cc:1397] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260809 00:52:14.714365 12874 tablet_bootstrap.cc:654] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:14.714536 12879 tablet_bootstrap.cc:492] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24: No bootstrap required, opened a new log
I20260809 00:52:14.714854 12879 ts_tablet_manager.cc:1397] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 00:52:14.715756 12874 log.cc:824] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:14.725369 12872 raft_consensus.cc:348] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } }
I20260809 00:52:14.725960 12872 raft_consensus.cc:374] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:14.726071 12872 raft_consensus.cc:729] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935d4267580747898906973bfbc9fd06, State: Initialized, Role: FOLLOWER
I20260809 00:52:14.726703 12879 raft_consensus.cc:348] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } }
I20260809 00:52:14.727304 12879 raft_consensus.cc:374] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:14.727077 12872 consensus_queue.cc:260] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } }
I20260809 00:52:14.727452 12879 raft_consensus.cc:729] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303026dfbcd044c0abe4bc0a11c92e24, State: Initialized, Role: FOLLOWER
I20260809 00:52:14.729053 12839 heartbeater.cc:503] Master 127.9.225.62:38157 was elected leader, sending a full tablet report...
I20260809 00:52:14.730209 12872 ts_tablet_manager.cc:1428] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260809 00:52:14.731175 12879 consensus_queue.cc:260] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } }
I20260809 00:52:14.731719 12874 tablet_bootstrap.cc:492] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999: No bootstrap required, opened a new log
I20260809 00:52:14.732137 12874 ts_tablet_manager.cc:1397] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.014s
I20260809 00:52:14.738160 12879 ts_tablet_manager.cc:1428] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24: Time spent starting tablet: real 0.023s user 0.021s sys 0.000s
I20260809 00:52:14.744187 12874 raft_consensus.cc:348] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } }
I20260809 00:52:14.744683 12874 raft_consensus.cc:374] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:14.744812 12874 raft_consensus.cc:729] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329d3b81bb7342f6a40618d09ccf8999, State: Initialized, Role: FOLLOWER
I20260809 00:52:14.745599 12874 consensus_queue.cc:260] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } }
I20260809 00:52:14.746803 12627 heartbeater.cc:503] Master 127.9.225.62:38157 was elected leader, sending a full tablet report...
I20260809 00:52:14.747596 12874 ts_tablet_manager.cc:1428] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260809 00:52:14.810446 12841 tablet.cc:2369] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:52:14.905295 12403 tablet.cc:2369] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:52:14.978772 12629 tablet.cc:2369] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:52:15.007650 12890 raft_consensus.cc:482] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:52:15.008037 12890 raft_consensus.cc:504] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } }
I20260809 00:52:15.009997 12890 leader_election.cc:288] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 935d4267580747898906973bfbc9fd06 (127.9.225.3:45767), 303026dfbcd044c0abe4bc0a11c92e24 (127.9.225.1:42655)
I20260809 00:52:15.016909 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.010488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:49981 (local address 127.9.225.3:45767)
0809 00:52:15.010681 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.010688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.011326 (+ 638us) server_negotiation.cc:407] Connection header received
0809 00:52:15.011591 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.011596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.011662 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.011766 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:15.012372 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.013113 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.015104 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.015878 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.016042 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.016160 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.016497 (+ 337us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.016635 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 00:52:15.017009 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.010483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:38685 (local address 127.9.225.1:42655)
0809 00:52:15.010697 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.010704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.011777 (+ 1073us) server_negotiation.cc:407] Connection header received
0809 00:52:15.011980 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.011984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.012044 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.012142 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:15.012700 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.013484 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.015238 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.016036 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.016199 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.016313 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.016625 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.016761 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 00:52:15.017091 12892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.010373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:45767 (local address 127.9.225.2:49981)
0809 00:52:15.011182 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:15.011201 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:15.011415 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:15.011789 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:15.011794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:15.011890 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:15.012199 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.012209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.013263 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.013270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.014914 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.014927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.016099 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.016105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.016294 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.016342 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 00:52:15.016459 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 00:52:15.016599 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":2,"spinlock_wait_cycles":25984,"thread_start_us":488,"threads_started":1}
I20260809 00:52:15.017170 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.010572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:42655 (local address 127.9.225.2:38685)
0809 00:52:15.011650 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:15.011665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:15.011827 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:15.012162 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:15.012169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:15.012198 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:15.012532 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.012540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.013618 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.013627 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.015067 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.015079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.016273 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.016280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.016463 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.016487 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 00:52:15.016594 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 00:52:15.016718 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":31,"thread_start_us":730,"threads_started":1}
I20260809 00:52:15.018150 12760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5749f59f614fbc9b0fa60ad5af2bac" candidate_uuid: "329d3b81bb7342f6a40618d09ccf8999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935d4267580747898906973bfbc9fd06" is_pre_election: true
I20260809 00:52:15.018146 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5749f59f614fbc9b0fa60ad5af2bac" candidate_uuid: "329d3b81bb7342f6a40618d09ccf8999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303026dfbcd044c0abe4bc0a11c92e24" is_pre_election: true
I20260809 00:52:15.018684 12760 raft_consensus.cc:2454] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 329d3b81bb7342f6a40618d09ccf8999 in term 0.
I20260809 00:52:15.018683 12319 raft_consensus.cc:2454] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 329d3b81bb7342f6a40618d09ccf8999 in term 0.
I20260809 00:52:15.019392 12437 leader_election.cc:302] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 303026dfbcd044c0abe4bc0a11c92e24, 329d3b81bb7342f6a40618d09ccf8999, 935d4267580747898906973bfbc9fd06; no voters:
I20260809 00:52:15.019816 12890 raft_consensus.cc:2781] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:52:15.019966 12890 raft_consensus.cc:482] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:52:15.020088 12890 raft_consensus.cc:3037] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:15.022405 12890 raft_consensus.cc:504] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } }
I20260809 00:52:15.023046 12890 leader_election.cc:288] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [CANDIDATE]: Term 1 election: Requested vote from peers 935d4267580747898906973bfbc9fd06 (127.9.225.3:45767), 303026dfbcd044c0abe4bc0a11c92e24 (127.9.225.1:42655)
I20260809 00:52:15.023648 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5749f59f614fbc9b0fa60ad5af2bac" candidate_uuid: "329d3b81bb7342f6a40618d09ccf8999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303026dfbcd044c0abe4bc0a11c92e24"
I20260809 00:52:15.023645 12760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c5749f59f614fbc9b0fa60ad5af2bac" candidate_uuid: "329d3b81bb7342f6a40618d09ccf8999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935d4267580747898906973bfbc9fd06"
I20260809 00:52:15.023979 12319 raft_consensus.cc:3037] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:15.024003 12760 raft_consensus.cc:3037] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:15.026342 12319 raft_consensus.cc:2454] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 329d3b81bb7342f6a40618d09ccf8999 in term 1.
I20260809 00:52:15.026589 12760 raft_consensus.cc:2454] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 329d3b81bb7342f6a40618d09ccf8999 in term 1.
I20260809 00:52:15.026872 12436 leader_election.cc:302] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 303026dfbcd044c0abe4bc0a11c92e24, 329d3b81bb7342f6a40618d09ccf8999; no voters:
I20260809 00:52:15.027249 12890 raft_consensus.cc:2781] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:15.027700 12890 raft_consensus.cc:686] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 1 LEADER]: Becoming Leader. State: Replica: 329d3b81bb7342f6a40618d09ccf8999, State: Running, Role: LEADER
I20260809 00:52:15.028288 12890 consensus_queue.cc:237] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } }
I20260809 00:52:15.033469 12046 catalog_manager.cc:5495] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 reported cstate change: term changed from 0 to 1, leader changed from <none> to 329d3b81bb7342f6a40618d09ccf8999 (127.9.225.2). New cstate: current_term: 1 leader_uuid: "329d3b81bb7342f6a40618d09ccf8999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } health_report { overall_health: UNKNOWN } } }
I20260809 00:52:15.070189 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.063940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46650 (local address 127.9.225.2:37587)
0809 00:52:15.064203 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.064211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.064239 (+ 28us) server_negotiation.cc:407] Connection header received
0809 00:52:15.064669 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.064680 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.064766 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.064935 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:15.065396 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.066149 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.067572 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.068064 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.068209 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.068770 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:15.069494 (+ 724us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:15.069663 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.069806 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.069896 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260809 00:52:15.084220 12319 raft_consensus.cc:1261] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24 [term 1 FOLLOWER]: Refusing update from remote peer 329d3b81bb7342f6a40618d09ccf8999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:15.085461 12894 consensus_queue.cc:1035] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:15.085875 12760 raft_consensus.cc:1261] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06 [term 1 FOLLOWER]: Refusing update from remote peer 329d3b81bb7342f6a40618d09ccf8999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:15.086885 12894 consensus_queue.cc:1035] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:15.098948 12897 mvcc.cc:205] Tried to move back new op lower bound from 7316425666874281984 to 7316425666680729600. Current Snapshot: MvccSnapshot[applied={T|T < 7316425666874281984}]
I20260809 00:52:15.125089 12048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:15.125553 12048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:15.130401 12048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50: Sending DeleteTablet for 3 replicas of tablet 4c5749f59f614fbc9b0fa60ad5af2bac
I20260809 00:52:15.131954 12518 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c5749f59f614fbc9b0fa60ad5af2bac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:15 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:46642
I20260809 00:52:15.133417 12284 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c5749f59f614fbc9b0fa60ad5af2bac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:15 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:42748
I20260809 00:52:15.133912 12736 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c5749f59f614fbc9b0fa60ad5af2bac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:15 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:57842
I20260809 00:52:15.138350 12048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:52:15.139286 12048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:52:15.143509 12924 tablet_replica.cc:330] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06: stopping tablet replica
I20260809 00:52:15.144229 12921 tablet_replica.cc:330] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24: stopping tablet replica
I20260809 00:52:15.145550 12924 raft_consensus.cc:2229] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:15.145874 12921 raft_consensus.cc:2229] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:15.146533 12921 raft_consensus.cc:2258] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:15.147107 12924 raft_consensus.cc:2258] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:15.150220 12921 ts_tablet_manager.cc:1905] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:15.152729 12736 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd4a6c1cfb6422eb17638658e8041dc (DEFAULT_TABLE table=test-table [id=f77e4c11b06145d49afe241090f65cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:15.153296 12736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd4a6c1cfb6422eb17638658e8041dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:15.153142 12516 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd4a6c1cfb6422eb17638658e8041dc (DEFAULT_TABLE table=test-table [id=f77e4c11b06145d49afe241090f65cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:15.153846 12516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd4a6c1cfb6422eb17638658e8041dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:15.157879 12284 tablet_service.cc:1467] Processing CreateTablet for tablet 2fd4a6c1cfb6422eb17638658e8041dc (DEFAULT_TABLE table=test-table [id=f77e4c11b06145d49afe241090f65cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:15.158397 12284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fd4a6c1cfb6422eb17638658e8041dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:15.162968 12924 ts_tablet_manager.cc:1905] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:15.163388 12874 tablet_bootstrap.cc:492] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999: Bootstrap starting.
I20260809 00:52:15.165320 12874 tablet_bootstrap.cc:654] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:15.170737 12879 tablet_bootstrap.cc:492] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24: Bootstrap starting.
I20260809 00:52:15.172508 12879 tablet_bootstrap.cc:654] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:15.169832 12872 tablet_bootstrap.cc:492] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06: Bootstrap starting.
I20260809 00:52:15.175405 12872 tablet_bootstrap.cc:654] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:15.178769 12872 tablet_bootstrap.cc:492] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06: No bootstrap required, opened a new log
I20260809 00:52:15.179000 12872 ts_tablet_manager.cc:1397] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20260809 00:52:15.179965 12872 raft_consensus.cc:348] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.180218 12872 raft_consensus.cc:374] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:15.180346 12872 raft_consensus.cc:729] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935d4267580747898906973bfbc9fd06, State: Initialized, Role: FOLLOWER
I20260809 00:52:15.180688 12872 consensus_queue.cc:260] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.181461 12872 ts_tablet_manager.cc:1428] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260809 00:52:15.186411 12917 tablet_replica.cc:330] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999: stopping tablet replica
I20260809 00:52:15.187240 12879 tablet_bootstrap.cc:492] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24: No bootstrap required, opened a new log
I20260809 00:52:15.187426 12879 ts_tablet_manager.cc:1397] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20260809 00:52:15.187690 12917 raft_consensus.cc:2229] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:52:15.188526 12879 raft_consensus.cc:348] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.188769 12879 raft_consensus.cc:374] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:15.188886 12879 raft_consensus.cc:729] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303026dfbcd044c0abe4bc0a11c92e24, State: Initialized, Role: FOLLOWER
I20260809 00:52:15.188862 12917 raft_consensus.cc:2258] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:15.189173 12874 tablet_bootstrap.cc:492] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999: No bootstrap required, opened a new log
I20260809 00:52:15.189422 12874 ts_tablet_manager.cc:1397] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.008s
I20260809 00:52:15.189467 12879 consensus_queue.cc:260] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.190052 12884 raft_consensus.cc:482] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:52:15.190266 12884 raft_consensus.cc:504] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.190502 12874 raft_consensus.cc:348] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.190892 12874 raft_consensus.cc:374] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:15.191058 12874 raft_consensus.cc:729] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329d3b81bb7342f6a40618d09ccf8999, State: Initialized, Role: FOLLOWER
I20260809 00:52:15.191148 12917 ts_tablet_manager.cc:1905] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:15.191497 12874 consensus_queue.cc:260] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.192400 12874 ts_tablet_manager.cc:1428] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 00:52:15.192363 12884 leader_election.cc:288] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 303026dfbcd044c0abe4bc0a11c92e24 (127.9.225.1:42655), 329d3b81bb7342f6a40618d09ccf8999 (127.9.225.2:37587)
I20260809 00:52:15.198830 12879 ts_tablet_manager.cc:1428] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260809 00:52:15.198691 12904 raft_consensus.cc:482] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:52:15.199065 12904 raft_consensus.cc:504] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.200124 12904 leader_election.cc:288] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 303026dfbcd044c0abe4bc0a11c92e24 (127.9.225.1:42655), 935d4267580747898906973bfbc9fd06 (127.9.225.3:45767)
I20260809 00:52:15.201514 12760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd4a6c1cfb6422eb17638658e8041dc" candidate_uuid: "329d3b81bb7342f6a40618d09ccf8999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935d4267580747898906973bfbc9fd06" is_pre_election: true
I20260809 00:52:15.201902 12760 raft_consensus.cc:2454] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 329d3b81bb7342f6a40618d09ccf8999 in term 0.
I20260809 00:52:15.202234 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd4a6c1cfb6422eb17638658e8041dc" candidate_uuid: "329d3b81bb7342f6a40618d09ccf8999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303026dfbcd044c0abe4bc0a11c92e24" is_pre_election: true
I20260809 00:52:15.202507 12319 raft_consensus.cc:2454] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 329d3b81bb7342f6a40618d09ccf8999 in term 0.
I20260809 00:52:15.202505 12437 leader_election.cc:302] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 329d3b81bb7342f6a40618d09ccf8999, 935d4267580747898906973bfbc9fd06; no voters:
I20260809 00:52:15.202929 12904 raft_consensus.cc:2781] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:52:15.203102 12904 raft_consensus.cc:482] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:52:15.203223 12904 raft_consensus.cc:3037] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:15.205277 12924 ts_tablet_manager.cc:1918] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:52:15.205613 12924 log.cc:1196] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/4c5749f59f614fbc9b0fa60ad5af2bac
I20260809 00:52:15.206344 12924 ts_tablet_manager.cc:1939] T 4c5749f59f614fbc9b0fa60ad5af2bac P 935d4267580747898906973bfbc9fd06: Deleting consensus metadata
I20260809 00:52:15.208629 12027 catalog_manager.cc:4841] TS 935d4267580747898906973bfbc9fd06 (127.9.225.3:45767): tablet 4c5749f59f614fbc9b0fa60ad5af2bac (table test-table [id=8eef11e24b7640dd86f424394576c6bb]) successfully deleted
I20260809 00:52:15.208747 12904 raft_consensus.cc:504] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.210124 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd4a6c1cfb6422eb17638658e8041dc" candidate_uuid: "329d3b81bb7342f6a40618d09ccf8999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303026dfbcd044c0abe4bc0a11c92e24"
I20260809 00:52:15.210282 12921 ts_tablet_manager.cc:1918] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:52:15.210386 12319 raft_consensus.cc:3037] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:15.210530 12921 log.cc:1196] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/4c5749f59f614fbc9b0fa60ad5af2bac
I20260809 00:52:15.211212 12921 ts_tablet_manager.cc:1939] T 4c5749f59f614fbc9b0fa60ad5af2bac P 303026dfbcd044c0abe4bc0a11c92e24: Deleting consensus metadata
I20260809 00:52:15.211572 12917 ts_tablet_manager.cc:1918] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:52:15.211735 12917 log.cc:1196] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/4c5749f59f614fbc9b0fa60ad5af2bac
I20260809 00:52:15.212252 12917 ts_tablet_manager.cc:1939] T 4c5749f59f614fbc9b0fa60ad5af2bac P 329d3b81bb7342f6a40618d09ccf8999: Deleting consensus metadata
I20260809 00:52:15.214334 12024 catalog_manager.cc:4841] TS 303026dfbcd044c0abe4bc0a11c92e24 (127.9.225.1:42655): tablet 4c5749f59f614fbc9b0fa60ad5af2bac (table test-table [id=8eef11e24b7640dd86f424394576c6bb]) successfully deleted
I20260809 00:52:15.214568 12022 catalog_manager.cc:4841] TS 329d3b81bb7342f6a40618d09ccf8999 (127.9.225.2:37587): tablet 4c5749f59f614fbc9b0fa60ad5af2bac (table test-table [id=8eef11e24b7640dd86f424394576c6bb]) successfully deleted
I20260809 00:52:15.214697 12904 leader_election.cc:288] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [CANDIDATE]: Term 1 election: Requested vote from peers 303026dfbcd044c0abe4bc0a11c92e24 (127.9.225.1:42655), 935d4267580747898906973bfbc9fd06 (127.9.225.3:45767)
I20260809 00:52:15.215222 12760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd4a6c1cfb6422eb17638658e8041dc" candidate_uuid: "329d3b81bb7342f6a40618d09ccf8999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935d4267580747898906973bfbc9fd06"
I20260809 00:52:15.215488 12760 raft_consensus.cc:3037] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:15.219756 12760 raft_consensus.cc:2454] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 329d3b81bb7342f6a40618d09ccf8999 in term 1.
I20260809 00:52:15.220327 12437 leader_election.cc:302] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 329d3b81bb7342f6a40618d09ccf8999, 935d4267580747898906973bfbc9fd06; no voters:
I20260809 00:52:15.225174 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.193283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:37767 (local address 127.9.225.2:37587)
0809 00:52:15.193545 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.193554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.203970 (+ 10416us) server_negotiation.cc:407] Connection header received
0809 00:52:15.204288 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.204299 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.206949 (+ 2650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.207065 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:15.216610 (+ 9545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.218399 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.221868 (+ 3469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.223377 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.223515 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.223620 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.224479 (+ 859us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.224689 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260809 00:52:15.225798 12319 raft_consensus.cc:2454] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 329d3b81bb7342f6a40618d09ccf8999 in term 1.
I20260809 00:52:15.226101 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.192685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:37587 (local address 127.9.225.3:37767)
0809 00:52:15.203852 (+ 11167us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:15.203879 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:15.204101 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:15.215585 (+ 11484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:15.215590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:15.215616 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:15.216305 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.216338 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.218607 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.218612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.221654 (+ 3042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.221672 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.223882 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.223893 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.224214 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.224279 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 00:52:15.225428 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0809 00:52:15.225618 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11007,"thread_start_us":20668,"threads_started":1}
I20260809 00:52:15.227221 12904 raft_consensus.cc:2781] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:15.227475 12904 raft_consensus.cc:686] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 1 LEADER]: Becoming Leader. State: Replica: 329d3b81bb7342f6a40618d09ccf8999, State: Running, Role: LEADER
I20260809 00:52:15.227872 12904 consensus_queue.cc:237] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.228163 12556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd4a6c1cfb6422eb17638658e8041dc" candidate_uuid: "935d4267580747898906973bfbc9fd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "329d3b81bb7342f6a40618d09ccf8999" is_pre_election: true
I20260809 00:52:15.231818 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.194215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:34461 (local address 127.9.225.1:42655)
0809 00:52:15.194467 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.194477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.213527 (+ 19050us) server_negotiation.cc:407] Connection header received
0809 00:52:15.213607 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.213612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.213699 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.213817 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:15.220681 (+ 6864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.222171 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.228858 (+ 6687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.230573 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.230706 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.230838 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.231316 (+ 478us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.231459 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 00:52:15.231863 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.192906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:42655 (local address 127.9.225.3:34461)
0809 00:52:15.209813 (+ 16907us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:15.209845 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:15.210085 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:15.214910 (+ 4825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:15.214915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:15.214976 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:15.215368 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.215383 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.224273 (+ 8890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.224278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.228580 (+ 4302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.228599 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.230968 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.230973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.231109 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.231134 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 00:52:15.231270 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 00:52:15.231436 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16714,"thread_start_us":10612,"threads_started":1}
I20260809 00:52:15.232525 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fd4a6c1cfb6422eb17638658e8041dc" candidate_uuid: "935d4267580747898906973bfbc9fd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303026dfbcd044c0abe4bc0a11c92e24" is_pre_election: true
I20260809 00:52:15.232996 12319 raft_consensus.cc:2379] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 935d4267580747898906973bfbc9fd06 in current term 1: Already voted for candidate 329d3b81bb7342f6a40618d09ccf8999 in this term.
I20260809 00:52:15.233827 12660 leader_election.cc:302] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 935d4267580747898906973bfbc9fd06; no voters: 303026dfbcd044c0abe4bc0a11c92e24, 329d3b81bb7342f6a40618d09ccf8999
I20260809 00:52:15.234890 12884 raft_consensus.cc:2726] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 00:52:15.234620 12048 catalog_manager.cc:5495] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 reported cstate change: term changed from 0 to 1, leader changed from <none> to 329d3b81bb7342f6a40618d09ccf8999 (127.9.225.2). New cstate: current_term: 1 leader_uuid: "329d3b81bb7342f6a40618d09ccf8999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } health_report { overall_health: UNKNOWN } } }
Aug 09 00:52:15 dist-test-slave-j2vl krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236732, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:15.324779 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.295709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:38157 (local address 127.0.0.1:58946)
0809 00:52:15.297250 (+ 1541us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:15.297280 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:15.297541 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:15.299599 (+ 2058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:15.299605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:15.300678 (+ 1073us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:15.301030 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.301045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.303408 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.303415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.305281 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.305302 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.306479 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.306487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.306695 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.310321 (+ 3626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:15.310351 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:15.314454 (+ 4103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:15.321203 (+ 6749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:15.321219 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:15.321227 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:15.322039 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:15.322582 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:15.322586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:15.322588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:15.322852 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:15.323637 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:15.323643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:15.323921 (+ 278us) client_negotiation.cc:769] Sending connection context
0809 00:52:15.324077 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 00:52:15.324299 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1248,"thread_start_us":865,"threads_started":1}
I20260809 00:52:15.325253 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.295627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.9.225.62:38157)
0809 00:52:15.299079 (+ 3452us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.299088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.299150 (+ 62us) server_negotiation.cc:407] Connection header received
0809 00:52:15.299225 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.299250 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.299361 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.299529 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:15.301255 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.303221 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.305498 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.306271 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.306382 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.314780 (+ 8398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:15.314790 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:15.314796 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:15.314849 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:15.320993 (+ 6144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:15.322220 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:15.322228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:15.322231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:15.322458 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:15.322984 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:15.322990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:15.322993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:15.323495 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:15.323744 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.324251 (+ 507us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.324519 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3166,"thread_start_us":2847,"threads_started":1}
I20260809 00:52:15.332597 12048 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:58946:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 00:52:15.334195 12048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:52:15.343169 12284 tablet_service.cc:1467] Processing CreateTablet for tablet 4631920433ba43d3a4c4cc8693e0a008 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50ddf4984ba446449fac0d3192e435c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 00:52:15.342989 12518 tablet_service.cc:1467] Processing CreateTablet for tablet 4631920433ba43d3a4c4cc8693e0a008 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50ddf4984ba446449fac0d3192e435c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 00:52:15.343571 12736 tablet_service.cc:1467] Processing CreateTablet for tablet 4631920433ba43d3a4c4cc8693e0a008 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50ddf4984ba446449fac0d3192e435c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 00:52:15.343972 12518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4631920433ba43d3a4c4cc8693e0a008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:15.344033 12284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4631920433ba43d3a4c4cc8693e0a008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:15.344318 12736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4631920433ba43d3a4c4cc8693e0a008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:15.355705 12874 tablet_bootstrap.cc:492] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999: Bootstrap starting.
I20260809 00:52:15.358194 12879 tablet_bootstrap.cc:492] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24: Bootstrap starting.
I20260809 00:52:15.359354 12872 tablet_bootstrap.cc:492] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: Bootstrap starting.
I20260809 00:52:15.362164 12879 tablet_bootstrap.cc:654] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:15.363879 12872 tablet_bootstrap.cc:654] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:15.366618 12874 tablet_bootstrap.cc:654] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:15.369446 12872 tablet_bootstrap.cc:492] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: No bootstrap required, opened a new log
I20260809 00:52:15.369856 12872 ts_tablet_manager.cc:1397] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260809 00:52:15.371508 12872 raft_consensus.cc:348] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.371901 12874 tablet_bootstrap.cc:492] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999: No bootstrap required, opened a new log
I20260809 00:52:15.371929 12872 raft_consensus.cc:374] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:15.372088 12872 raft_consensus.cc:729] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935d4267580747898906973bfbc9fd06, State: Initialized, Role: FOLLOWER
I20260809 00:52:15.372268 12874 ts_tablet_manager.cc:1397] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20260809 00:52:15.372301 12879 tablet_bootstrap.cc:492] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24: No bootstrap required, opened a new log
I20260809 00:52:15.372828 12879 ts_tablet_manager.cc:1397] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260809 00:52:15.372633 12872 consensus_queue.cc:260] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.373505 12884 tablet_replica.cc:439] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } }
I20260809 00:52:15.373989 12884 tablet_replica.cc:442] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:15.373883 12874 raft_consensus.cc:348] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.374076 12879 raft_consensus.cc:348] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.374327 12874 raft_consensus.cc:374] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:15.374423 12879 raft_consensus.cc:374] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:15.374485 12874 raft_consensus.cc:729] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329d3b81bb7342f6a40618d09ccf8999, State: Initialized, Role: FOLLOWER
I20260809 00:52:15.374541 12879 raft_consensus.cc:729] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303026dfbcd044c0abe4bc0a11c92e24, State: Initialized, Role: FOLLOWER
I20260809 00:52:15.375000 12874 consensus_queue.cc:260] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.375134 12879 consensus_queue.cc:260] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.375949 12904 tablet_replica.cc:439] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } }
I20260809 00:52:15.375742 12885 tablet_replica.cc:439] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } }
I20260809 00:52:15.376480 12904 tablet_replica.cc:442] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:15.376506 12885 tablet_replica.cc:442] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:15.377961 12874 ts_tablet_manager.cc:1428] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 00:52:15.379909 12879 ts_tablet_manager.cc:1428] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260809 00:52:15.381083 12884 raft_consensus.cc:482] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:52:15.381291 12884 raft_consensus.cc:504] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.382345 12884 leader_election.cc:288] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 303026dfbcd044c0abe4bc0a11c92e24 (127.9.225.1:42655), 329d3b81bb7342f6a40618d09ccf8999 (127.9.225.2:37587)
I20260809 00:52:15.382850 12556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4631920433ba43d3a4c4cc8693e0a008" candidate_uuid: "935d4267580747898906973bfbc9fd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "329d3b81bb7342f6a40618d09ccf8999" is_pre_election: true
I20260809 00:52:15.383069 12872 ts_tablet_manager.cc:1428] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: Time spent starting tablet: real 0.013s user 0.000s sys 0.004s
I20260809 00:52:15.383208 12556 raft_consensus.cc:2454] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 935d4267580747898906973bfbc9fd06 in term 0.
I20260809 00:52:15.383963 12658 leader_election.cc:302] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 329d3b81bb7342f6a40618d09ccf8999, 935d4267580747898906973bfbc9fd06; no voters:
I20260809 00:52:15.384286 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4631920433ba43d3a4c4cc8693e0a008" candidate_uuid: "935d4267580747898906973bfbc9fd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303026dfbcd044c0abe4bc0a11c92e24" is_pre_election: true
I20260809 00:52:15.384742 12319 raft_consensus.cc:2454] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 935d4267580747898906973bfbc9fd06 in term 0.
I20260809 00:52:15.384780 12884 raft_consensus.cc:2781] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:52:15.385011 12884 raft_consensus.cc:482] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:52:15.385210 12884 raft_consensus.cc:3037] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:15.390877 12884 raft_consensus.cc:504] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.392091 12884 leader_election.cc:288] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [CANDIDATE]: Term 1 election: Requested vote from peers 303026dfbcd044c0abe4bc0a11c92e24 (127.9.225.1:42655), 329d3b81bb7342f6a40618d09ccf8999 (127.9.225.2:37587)
I20260809 00:52:15.392901 12556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4631920433ba43d3a4c4cc8693e0a008" candidate_uuid: "935d4267580747898906973bfbc9fd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "329d3b81bb7342f6a40618d09ccf8999"
I20260809 00:52:15.392858 12319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4631920433ba43d3a4c4cc8693e0a008" candidate_uuid: "935d4267580747898906973bfbc9fd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303026dfbcd044c0abe4bc0a11c92e24"
I20260809 00:52:15.393383 12556 raft_consensus.cc:3037] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:15.393445 12319 raft_consensus.cc:3037] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:15.398002 12319 raft_consensus.cc:2454] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 935d4267580747898906973bfbc9fd06 in term 1.
I20260809 00:52:15.398123 12556 raft_consensus.cc:2454] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 935d4267580747898906973bfbc9fd06 in term 1.
I20260809 00:52:15.398844 12660 leader_election.cc:302] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 303026dfbcd044c0abe4bc0a11c92e24, 935d4267580747898906973bfbc9fd06; no voters:
I20260809 00:52:15.399308 12884 raft_consensus.cc:2781] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:15.400485 12884 raft_consensus.cc:686] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [term 1 LEADER]: Becoming Leader. State: Replica: 935d4267580747898906973bfbc9fd06, State: Running, Role: LEADER
I20260809 00:52:15.401252 12884 consensus_queue.cc:237] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } }
I20260809 00:52:15.407305 12950 tablet_replica.cc:439] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: TxnStatusTablet state changed. Reason: New leader 935d4267580747898906973bfbc9fd06. Latest consensus state: current_term: 1 leader_uuid: "935d4267580747898906973bfbc9fd06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } }
I20260809 00:52:15.408092 12950 tablet_replica.cc:442] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: This TxnStatusTablet replica's current role is: LEADER
I20260809 00:52:15.410038 12954 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 00:52:15.410584 12954 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 00:52:15.410936 12045 catalog_manager.cc:5495] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 935d4267580747898906973bfbc9fd06 (127.9.225.3). New cstate: current_term: 1 leader_uuid: "935d4267580747898906973bfbc9fd06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } health_report { overall_health: HEALTHY } } }
I20260809 00:52:15.448777 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.440159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.9.225.3:45767)
0809 00:52:15.441002 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.441010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.441031 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:52:15.441198 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.441204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.441471 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.441652 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:15.442367 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.443531 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.445743 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.446417 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.446597 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.447323 (+ 726us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:15.447939 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:15.448105 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.448230 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.448321 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669}
I20260809 00:52:15.448961 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.440127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:45767 (local address 127.0.0.1:45654)
0809 00:52:15.440537 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:15.440564 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:15.440692 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:15.441880 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:15.441883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:15.441893 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 00:52:15.442195 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.442223 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.443676 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.443682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.445216 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.445230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.446662 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.446669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.446976 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.447048 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 00:52:15.448061 (+ 1013us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 00:52:15.448088 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 00:52:15.448207 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 00:52:15.448429 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260809 00:52:15.455550 12319 raft_consensus.cc:1261] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24 [term 1 FOLLOWER]: Refusing update from remote peer 935d4267580747898906973bfbc9fd06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:15.455716 12556 raft_consensus.cc:1261] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999 [term 1 FOLLOWER]: Refusing update from remote peer 935d4267580747898906973bfbc9fd06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:15.456782 12884 consensus_queue.cc:1035] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:15.457649 12950 consensus_queue.cc:1035] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:15.472720 12885 tablet_replica.cc:439] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24: TxnStatusTablet state changed. Reason: New leader 935d4267580747898906973bfbc9fd06. Latest consensus state: current_term: 1 leader_uuid: "935d4267580747898906973bfbc9fd06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } }
I20260809 00:52:15.473414 12885 tablet_replica.cc:442] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:15.479825 12884 tablet_replica.cc:439] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "935d4267580747898906973bfbc9fd06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } }
I20260809 00:52:15.480646 12884 tablet_replica.cc:442] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: This TxnStatusTablet replica's current role is: LEADER
I20260809 00:52:15.481063 12899 mvcc.cc:205] Tried to move back new op lower bound from 7316425668413779968 to 7316425668214251520. Current Snapshot: MvccSnapshot[applied={T|T < 7316425668413779968}]
I20260809 00:52:15.483054 12898 mvcc.cc:205] Tried to move back new op lower bound from 7316425668413779968 to 7316425668214251520. Current Snapshot: MvccSnapshot[applied={T|T < 7316425668413779968}]
I20260809 00:52:15.483632 12885 tablet_replica.cc:439] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "935d4267580747898906973bfbc9fd06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } }
I20260809 00:52:15.484059 12885 tablet_replica.cc:442] T 4631920433ba43d3a4c4cc8693e0a008 P 303026dfbcd044c0abe4bc0a11c92e24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:15.490854 12904 tablet_replica.cc:439] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999: TxnStatusTablet state changed. Reason: New leader 935d4267580747898906973bfbc9fd06. Latest consensus state: current_term: 1 leader_uuid: "935d4267580747898906973bfbc9fd06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } }
I20260809 00:52:15.491513 12904 tablet_replica.cc:442] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:15.491663 12884 tablet_replica.cc:439] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "935d4267580747898906973bfbc9fd06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } }
I20260809 00:52:15.492096 12884 tablet_replica.cc:442] T 4631920433ba43d3a4c4cc8693e0a008 P 935d4267580747898906973bfbc9fd06: This TxnStatusTablet replica's current role is: LEADER
I20260809 00:52:15.500356 12904 tablet_replica.cc:439] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "935d4267580747898906973bfbc9fd06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 } } peers { permanent_uuid: "329d3b81bb7342f6a40618d09ccf8999" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37587 } } peers { permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 } } }
I20260809 00:52:15.500857 12904 tablet_replica.cc:442] T 4631920433ba43d3a4c4cc8693e0a008 P 329d3b81bb7342f6a40618d09ccf8999: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:15.524508 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.515292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:46547 (local address 127.9.225.3:45767)
0809 00:52:15.515649 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.515655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.518217 (+ 2562us) server_negotiation.cc:407] Connection header received
0809 00:52:15.518509 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.518514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.518594 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.518694 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:15.519222 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.519996 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.521950 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.522755 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.522918 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.523563 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:15.523674 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:15.523780 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.523929 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.524033 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260809 00:52:15.524679 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.515710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:45767 (local address 127.9.225.2:46547)
0809 00:52:15.518085 (+ 2375us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:15.518113 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:15.518308 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:15.518710 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:15.518715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:15.518728 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 00:52:15.519076 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.519085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.520194 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.520201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.521799 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.521811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.523015 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.523025 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.523274 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.523337 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 00:52:15.523785 (+ 448us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 00:52:15.523799 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 00:52:15.523905 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 00:52:15.524103 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2020,"thread_start_us":1422,"threads_started":1}
I20260809 00:52:15.540019 12760 raft_consensus.cc:1261] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [term 1 FOLLOWER]: Refusing update from remote peer 329d3b81bb7342f6a40618d09ccf8999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:15.540879 12319 raft_consensus.cc:1261] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24 [term 1 FOLLOWER]: Refusing update from remote peer 329d3b81bb7342f6a40618d09ccf8999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:15.541841 12890 consensus_queue.cc:1035] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "303026dfbcd044c0abe4bc0a11c92e24" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:15.543323 12890 consensus_queue.cc:1035] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "935d4267580747898906973bfbc9fd06" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:15.649232 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.623238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:51759 (local address 127.9.225.2:37587)
0809 00:52:15.623518 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.623530 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.635791 (+ 12261us) server_negotiation.cc:407] Connection header received
0809 00:52:15.636511 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.636518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.636658 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.636742 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:15.638849 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.640683 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.645095 (+ 4412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.646528 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.646683 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.648005 (+ 1322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:15.648219 (+ 214us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:15.648330 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.648656 (+ 326us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.648791 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 00:52:15.651065 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.623785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:37587 (local address 127.9.225.3:51759)
0809 00:52:15.635641 (+ 11856us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:15.635666 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:15.636256 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:15.637072 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:15.637076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:15.637102 (+ 26us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 00:52:15.638509 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.638527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.640915 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.640921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.644844 (+ 3923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:15.644864 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.647139 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.647150 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:15.647516 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.647648 (+ 132us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 00:52:15.648434 (+ 786us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 00:52:15.648461 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 00:52:15.649586 (+ 1125us) client_negotiation.cc:240] Negotiation successful
0809 00:52:15.649840 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11467,"thread_start_us":12042,"threads_started":1}
I20260809 00:52:15.734010 12048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:15.734310 12048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:15.738513 12048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 566efdc6fe83418896302d5d26098e50: Sending DeleteTablet for 3 replicas of tablet 2fd4a6c1cfb6422eb17638658e8041dc
I20260809 00:52:15.739778 12736 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fd4a6c1cfb6422eb17638658e8041dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:15 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:57842
I20260809 00:52:15.740355 12284 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fd4a6c1cfb6422eb17638658e8041dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:15 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:42748
I20260809 00:52:15.740569 12518 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fd4a6c1cfb6422eb17638658e8041dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:15 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:46642
I20260809 00:52:15.761900 12979 tablet_replica.cc:330] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24: stopping tablet replica
I20260809 00:52:15.763820 12979 raft_consensus.cc:2229] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:15.764336 12979 raft_consensus.cc:2258] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:15.765733 12978 tablet_replica.cc:330] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06: stopping tablet replica
I20260809 00:52:15.767679 12978 raft_consensus.cc:2229] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:15.768103 12978 raft_consensus.cc:2258] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:15.768114 12979 ts_tablet_manager.cc:1905] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:15.769582 12978 ts_tablet_manager.cc:1905] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:15.779197 12980 tablet_replica.cc:330] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999: stopping tablet replica
I20260809 00:52:15.779481 12980 raft_consensus.cc:2229] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:52:15.780203 12980 raft_consensus.cc:2258] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:15.781558 12980 ts_tablet_manager.cc:1905] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:15.782907 12978 ts_tablet_manager.cc:1918] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:52:15.783185 12978 log.cc:1196] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/2fd4a6c1cfb6422eb17638658e8041dc
I20260809 00:52:15.786804 12978 ts_tablet_manager.cc:1939] T 2fd4a6c1cfb6422eb17638658e8041dc P 935d4267580747898906973bfbc9fd06: Deleting consensus metadata
I20260809 00:52:15.788687 12027 catalog_manager.cc:4841] TS 935d4267580747898906973bfbc9fd06 (127.9.225.3:45767): tablet 2fd4a6c1cfb6422eb17638658e8041dc (table test-table [id=f77e4c11b06145d49afe241090f65cb8]) successfully deleted
Aug 09 00:52:15 dist-test-slave-j2vl krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:15.803212 12980 ts_tablet_manager.cc:1918] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:52:15.803252 12979 ts_tablet_manager.cc:1918] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:52:15.803426 12979 log.cc:1196] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/2fd4a6c1cfb6422eb17638658e8041dc
I20260809 00:52:15.803426 12980 log.cc:1196] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/2fd4a6c1cfb6422eb17638658e8041dc
I20260809 00:52:15.804153 12980 ts_tablet_manager.cc:1939] T 2fd4a6c1cfb6422eb17638658e8041dc P 329d3b81bb7342f6a40618d09ccf8999: Deleting consensus metadata
I20260809 00:52:15.805845 12022 catalog_manager.cc:4841] TS 329d3b81bb7342f6a40618d09ccf8999 (127.9.225.2:37587): tablet 2fd4a6c1cfb6422eb17638658e8041dc (table test-table [id=f77e4c11b06145d49afe241090f65cb8]) successfully deleted
I20260809 00:52:15.806389 12979 ts_tablet_manager.cc:1939] T 2fd4a6c1cfb6422eb17638658e8041dc P 303026dfbcd044c0abe4bc0a11c92e24: Deleting consensus metadata
I20260809 00:52:15.807994 12024 catalog_manager.cc:4841] TS 303026dfbcd044c0abe4bc0a11c92e24 (127.9.225.1:42655): tablet 2fd4a6c1cfb6422eb17638658e8041dc (table test-table [id=f77e4c11b06145d49afe241090f65cb8]) successfully deleted
I20260809 00:52:15.814114 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.767173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.9.225.1:42655)
0809 00:52:15.774314 (+ 7141us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.774325 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.775765 (+ 1440us) server_negotiation.cc:407] Connection header received
0809 00:52:15.775849 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.775854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.775978 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.776096 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:15.786717 (+ 10621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.788421 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.791259 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.792302 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.792446 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.801651 (+ 9205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:15.801674 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:15.801698 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:15.801753 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:15.808916 (+ 7163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:15.810617 (+ 1701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:15.810630 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:15.810634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:15.810950 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:15.811535 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:15.811542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:15.811546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:15.812259 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:15.812676 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.813071 (+ 395us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.813474 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6895,"thread_start_us":10159,"threads_started":1}
W20260809 00:52:15.815766 12388 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34280
I20260809 00:52:15.846498 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.820948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.9.225.62:38157)
0809 00:52:15.821184 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.821190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.821961 (+ 771us) server_negotiation.cc:407] Connection header received
0809 00:52:15.822154 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.822160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.822272 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.822444 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:15.825176 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.826722 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.829133 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.830203 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.830325 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.835669 (+ 5344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:15.835681 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:15.835688 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:15.835749 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:15.841601 (+ 5852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:15.842954 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:15.842961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:15.842965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:15.843238 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:15.843876 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:15.843883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:15.843886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:15.844449 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:15.844781 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.845449 (+ 668us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.845964 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260809 00:52:15.847429 12048 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58952
I20260809 00:52:15.850080 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12169
I20260809 00:52:15.874266 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.852769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.9.225.1:42655)
0809 00:52:15.853047 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.853053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.853091 (+ 38us) server_negotiation.cc:407] Connection header received
0809 00:52:15.853159 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.853166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.853248 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.853540 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:15.855339 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.856870 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.859042 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.859985 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.860138 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.863716 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:15.863724 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:15.863729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:15.863813 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:15.870240 (+ 6427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:15.871318 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:15.871325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:15.871328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:15.871553 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:15.872125 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:15.872132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:15.872137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:15.872678 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:15.873044 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.873433 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.873759 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260809 00:52:15.875633 12388 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34290
W20260809 00:52:15.876700 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12169: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:52:15.877156 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 12169
I20260809 00:52:15.919168 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12406
Aug 09 00:52:15 dist-test-slave-j2vl krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:52:15.942639 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.920293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.9.225.2:37587)
0809 00:52:15.920535 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.920542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.920565 (+ 23us) server_negotiation.cc:407] Connection header received
0809 00:52:15.920643 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.920648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.920742 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.920870 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:15.922679 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.923868 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.925879 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.926869 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.927052 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:15.933667 (+ 6615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:15.933690 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:15.933718 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:15.933781 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:15.938637 (+ 4856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:15.939851 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:15.939862 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:15.939866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:15.940056 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:15.940600 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:15.940605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:15.940609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:15.941132 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:15.941661 (+ 529us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:15.941962 (+ 301us) server_negotiation.cc:299] Negotiation successful
0809 00:52:15.942252 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260809 00:52:15.943523 12621 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41918
W20260809 00:52:15.944329 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12406: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:52:15.944532 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 12406
I20260809 00:52:15.993147 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12634
Aug 09 00:52:16 dist-test-slave-j2vl krb5kdc[11968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236734, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:52:16.010928 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:15.994107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.9.225.3:45767)
0809 00:52:15.994346 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:15.994352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:15.994374 (+ 22us) server_negotiation.cc:407] Connection header received
0809 00:52:15.994573 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:15.994578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:15.994645 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:15.994789 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:15.996216 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.997060 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.998451 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:15.998950 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:15.999098 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:16.003763 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:16.003777 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:16.003795 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:16.003836 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:16.007628 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:16.008482 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:16.008491 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:16.008494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:16.008663 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:16.009098 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:16.009105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:16.009114 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:16.009630 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:16.009999 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:16.010290 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 00:52:16.010570 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260809 00:52:16.011653 12833 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45656
W20260809 00:52:16.012436 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12634: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:52:16.012600 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 12634
I20260809 00:52:16.042387 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11992
I20260809 00:52:16.043696 12113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:16.162858 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 11992
2026-08-09T00:52:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5478 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:52:16 dist-test-slave-j2vl krb5kdc[12998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:52:16 dist-test-slave-j2vl krb5kdc[12998](info): set up 2 sockets
Aug 09 00:52:16 dist-test-slave-j2vl krb5kdc[12998](info): commencing operation
krb5kdc: starting...
W20260809 00:52:17.515810 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.303s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:52:17 dist-test-slave-j2vl krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:52:17.765190 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:44933
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:44933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:18.190011 13019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:18.190379 13019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:18.190481 13019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:18.198328 13019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:18.198477 13019 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 00:52:18.198520 13019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:18.198576 13019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:18.198632 13019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:18.198697 13019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:52:18.211097 13019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:44933
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:44933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:18.213505 13019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:18.215144 13019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:18.229923 13026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:18.229859 13029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:18.229876 13027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:18.235802 13019 server_base.cc:1034] running on GCE node
I20260809 00:52:18.236931 13019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:18.238878 13019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:18.240193 13019 hybrid_clock.cc:648] HybridClock initialized: now 1786236738240102 us; error 60 us; skew 500 ppm
I20260809 00:52:18.240694 13019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:18 dist-test-slave-j2vl krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236738, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:18.245436 13019 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:52:18.247915 13019 webserver.cc:460] Webserver started at http://127.9.225.62:42197/ using document root <none> and password file <none>
I20260809 00:52:18.248996 13019 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:18.249189 13019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:18.250034 13019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:18.254673 13019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "fa5b03483bdd438f92f0915000f86866"
format_stamp: "Formatted at 2026-08-09 00:52:18 on dist-test-slave-j2vl"
server_key: "be57a64ae02580196fb15b0a6ef309d4"
server_key_iv: "140308c24deb04e69886e7a3350ca4f0"
server_key_version: "encryptionkey@0"
I20260809 00:52:18.255977 13019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "fa5b03483bdd438f92f0915000f86866"
format_stamp: "Formatted at 2026-08-09 00:52:18 on dist-test-slave-j2vl"
server_key: "be57a64ae02580196fb15b0a6ef309d4"
server_key_iv: "140308c24deb04e69886e7a3350ca4f0"
server_key_version: "encryptionkey@0"
I20260809 00:52:18.262814 13019 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260809 00:52:18.267652 13043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:18.270277 13019 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260809 00:52:18.270656 13019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "fa5b03483bdd438f92f0915000f86866"
format_stamp: "Formatted at 2026-08-09 00:52:18 on dist-test-slave-j2vl"
server_key: "be57a64ae02580196fb15b0a6ef309d4"
server_key_iv: "140308c24deb04e69886e7a3350ca4f0"
server_key_version: "encryptionkey@0"
I20260809 00:52:18.271175 13019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:18.288269 13019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:18.294471 13019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:18.294842 13019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:18.319027 13019 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:44933
I20260809 00:52:18.319125 13155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:44933 every 8 connection(s)
I20260809 00:52:18.320611 13019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:18.324965 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 13019
I20260809 00:52:18.325222 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:18.325412 13156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:18.326022 10116 external_mini_cluster.cc:1428] Setting key 947d8c60ca0faa33459b712044d923fe
I20260809 00:52:18.342265 13156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866: Bootstrap starting.
Aug 09 00:52:18 dist-test-slave-j2vl krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:18.346432 13156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:18.348114 13156 log.cc:824] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:18.352314 13156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866: No bootstrap required, opened a new log
I20260809 00:52:18.355406 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:18.328424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.9.225.62:44933)
0809 00:52:18.329382 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:18.329398 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:18.329471 (+ 73us) server_negotiation.cc:407] Connection header received
0809 00:52:18.331706 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:18.331752 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:18.332463 (+ 711us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:18.333056 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:18.334965 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:18.336981 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:18.338552 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:18.339807 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:18.340032 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:18.345141 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:18.345154 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:18.345178 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:18.345251 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:18.349604 (+ 4353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:18.350513 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:18.350525 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:18.350535 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:18.350807 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:18.351348 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:18.351353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:18.351356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:18.352210 (+ 854us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:18.352573 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:18.353168 (+ 595us) server_negotiation.cc:299] Negotiation successful
0809 00:52:18.353745 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":402,"threads_started":1}
I20260809 00:52:18.362673 13156 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5b03483bdd438f92f0915000f86866" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44933 } }
I20260809 00:52:18.363125 13156 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:18.363207 13156 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5b03483bdd438f92f0915000f86866, State: Initialized, Role: FOLLOWER
I20260809 00:52:18.363878 13156 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5b03483bdd438f92f0915000f86866" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44933 } }
I20260809 00:52:18.364156 13156 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:18.364323 13156 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:18.364537 13156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:18.367007 13156 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5b03483bdd438f92f0915000f86866" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44933 } }
I20260809 00:52:18.367624 13156 leader_election.cc:302] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa5b03483bdd438f92f0915000f86866; no voters:
I20260809 00:52:18.368371 13156 leader_election.cc:288] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:18.368597 13168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:18.369465 13168 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [term 1 LEADER]: Becoming Leader. State: Replica: fa5b03483bdd438f92f0915000f86866, State: Running, Role: LEADER
I20260809 00:52:18.370095 13168 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5b03483bdd438f92f0915000f86866" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44933 } }
I20260809 00:52:18.371204 13156 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:18.373421 13169 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa5b03483bdd438f92f0915000f86866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5b03483bdd438f92f0915000f86866" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44933 } } }
I20260809 00:52:18.373895 13169 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:18.373860 13170 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa5b03483bdd438f92f0915000f86866. Latest consensus state: current_term: 1 leader_uuid: "fa5b03483bdd438f92f0915000f86866" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5b03483bdd438f92f0915000f86866" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44933 } } }
I20260809 00:52:18.374251 13170 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:18.377228 13183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:18.382829 13183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:18.395246 13183 catalog_manager.cc:1347] Generated new cluster ID: 3a505211f2ad4b00ad8929d719dbe9bb
I20260809 00:52:18.395395 13183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:18.409418 13183 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:18.411176 13183 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:18.421875 13183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866: Generated new TSK 0
I20260809 00:52:18.423269 13183 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:52:18.494956 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44933
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:18.887974 13201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:18.888338 13201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:18.888545 13201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:18.897053 13201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:18.897219 13201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:18.897436 13201 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 00:52:18.897549 13201 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 00:52:18.897698 13201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:52:18.908445 13201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.225.62:44933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:18.910049 13201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:18.911607 13201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:18.927450 13208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:18.927405 13212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:18.927505 13207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:18.931134 13201 server_base.cc:1034] running on GCE node
I20260809 00:52:18.931907 13201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:18.933038 13201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:18.934333 13201 hybrid_clock.cc:648] HybridClock initialized: now 1786236738934302 us; error 49 us; skew 500 ppm
I20260809 00:52:18.934803 13201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:18 dist-test-slave-j2vl krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236738, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:18.938670 13201 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:52:18.940491 13201 webserver.cc:460] Webserver started at http://127.9.225.1:40005/ using document root <none> and password file <none>
I20260809 00:52:18.941452 13201 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:18.941579 13201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:18.941892 13201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:18.945329 13201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "c1a6f3c288b04133af66a94d252e8ced"
format_stamp: "Formatted at 2026-08-09 00:52:18 on dist-test-slave-j2vl"
server_key: "9fb3101702c17deb88d52ff671092d6f"
server_key_iv: "f7bacb7dc69a61b94eb8777fa2d5021f"
server_key_version: "encryptionkey@0"
I20260809 00:52:18.946328 13201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "c1a6f3c288b04133af66a94d252e8ced"
format_stamp: "Formatted at 2026-08-09 00:52:18 on dist-test-slave-j2vl"
server_key: "9fb3101702c17deb88d52ff671092d6f"
server_key_iv: "f7bacb7dc69a61b94eb8777fa2d5021f"
server_key_version: "encryptionkey@0"
I20260809 00:52:18.952797 13201 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:52:18.957237 13224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:18.959021 13201 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260809 00:52:18.959268 13201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "c1a6f3c288b04133af66a94d252e8ced"
format_stamp: "Formatted at 2026-08-09 00:52:18 on dist-test-slave-j2vl"
server_key: "9fb3101702c17deb88d52ff671092d6f"
server_key_iv: "f7bacb7dc69a61b94eb8777fa2d5021f"
server_key_version: "encryptionkey@0"
I20260809 00:52:18.959520 13201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:18.977022 13201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:18.982223 13201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:18.982568 13201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:18.986187 13201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:18.986353 13201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:18.986508 13201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:18.986600 13201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 00:52:19 dist-test-slave-j2vl krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236738, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:19.016992 13236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:18.986703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:53105 (local address 127.9.225.62:44933)
0809 00:52:18.987717 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:18.987721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:18.988672 (+ 951us) server_negotiation.cc:407] Connection header received
0809 00:52:18.991280 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:18.991286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:18.991406 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:18.991552 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:18.994729 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:18.995904 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:18.998875 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:18.999609 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:18.999798 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:19.006938 (+ 7140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:19.006950 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:19.006958 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:19.007020 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:19.011672 (+ 4652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:19.013443 (+ 1771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:19.013452 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:19.013459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:19.013696 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:19.014522 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:19.014528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:19.014530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:19.015108 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:19.015305 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:19.016230 (+ 925us) server_negotiation.cc:299] Negotiation successful
0809 00:52:19.016533 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":728,"threads_started":1}
I20260809 00:52:19.018911 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:18.987505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44933 (local address 127.9.225.1:53105)
0809 00:52:18.988466 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:18.988523 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:18.990883 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:18.991916 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:18.991930 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:18.993076 (+ 1146us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:18.994414 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:18.994471 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:18.996141 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:18.996150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:18.998653 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:18.998673 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:18.999962 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:18.999980 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:19.000407 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:19.001618 (+ 1211us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:19.001676 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:19.006429 (+ 4753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:19.012021 (+ 5592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:19.012038 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:19.012074 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:19.013167 (+ 1093us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:19.013906 (+ 739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:19.013914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:19.013921 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:19.014336 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:19.015291 (+ 955us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:19.015302 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:19.015764 (+ 462us) client_negotiation.cc:769] Sending connection context
0809 00:52:19.016088 (+ 324us) client_negotiation.cc:240] Negotiation successful
0809 00:52:19.016764 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":424,"threads_started":1}
I20260809 00:52:19.041785 13201 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:32995
I20260809 00:52:19.041887 13413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:32995 every 8 connection(s)
I20260809 00:52:19.043391 13236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:19.026822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:38955 (local address 127.9.225.62:44933)
0809 00:52:19.027042 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:19.027047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:19.027955 (+ 908us) server_negotiation.cc:407] Connection header received
0809 00:52:19.028233 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:19.028238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:19.028313 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:19.028439 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:19.029605 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:19.030794 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:19.033288 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:19.034089 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:19.034247 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:19.036309 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:19.036315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:19.036318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:19.036357 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:19.039751 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:19.040629 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:19.040636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:19.040639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:19.040848 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:19.041577 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:19.041585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:19.041591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:19.042099 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:19.042258 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:19.042838 (+ 580us) server_negotiation.cc:299] Negotiation successful
0809 00:52:19.043124 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 00:52:19.043574 13378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:19.027091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44933 (local address 127.9.225.1:38955)
0809 00:52:19.027783 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:19.027803 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:19.028069 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:19.028476 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:19.028481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:19.029021 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:19.029420 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:19.029444 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:19.031320 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:19.031330 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:19.033085 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:19.033106 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:19.034335 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:19.034343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:19.034518 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:19.035263 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:19.035283 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:19.036068 (+ 785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:19.039945 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:19.039953 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:19.039957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:19.040458 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:19.041077 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:19.041087 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:19.041094 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:19.041318 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:19.042245 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:19.042251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:19.042491 (+ 240us) client_negotiation.cc:769] Sending connection context
0809 00:52:19.042669 (+ 178us) client_negotiation.cc:240] Negotiation successful
0809 00:52:19.043099 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":399,"threads_started":1}
I20260809 00:52:19.043715 13201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:52:19.046273 13414 heartbeater.cc:347] Connected to a master server at 127.9.225.62:44933
I20260809 00:52:19.046576 13414 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:19.047384 13414 heartbeater.cc:511] Master 127.9.225.62:44933 requested a full tablet report, sending...
I20260809 00:52:19.049844 13082 ts_manager.cc:194] Registered new tserver with Master: c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995)
I20260809 00:52:19.051473 13082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:53105
I20260809 00:52:19.053753 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 13201
I20260809 00:52:19.054011 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:52:19.054700 10116 external_mini_cluster.cc:1428] Setting key b5993a3d28eb57c1a2ff05dc5b230745
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:52:19.113941 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44933
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:19.508576 13421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:19.509306 13421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:19.509593 13421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:19.518517 13421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:19.518728 13421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:19.518903 13421 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 00:52:19.519023 13421 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 00:52:19.519160 13421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:52:19.531221 13421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.225.62:44933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:19.532709 13421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:19.534277 13421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:19.549398 13435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:19.549398 13428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:19.549716 13427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:19.550151 13421 server_base.cc:1034] running on GCE node
I20260809 00:52:19.551069 13421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:19.552170 13421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:19.553481 13421 hybrid_clock.cc:648] HybridClock initialized: now 1786236739553430 us; error 43 us; skew 500 ppm
I20260809 00:52:19.553886 13421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:19 dist-test-slave-j2vl krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236739, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:19.557571 13421 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:52:19.559369 13421 webserver.cc:460] Webserver started at http://127.9.225.2:35915/ using document root <none> and password file <none>
I20260809 00:52:19.560191 13421 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:19.560309 13421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:19.560609 13421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:19.564201 13421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "80d1702c61c443aaa1a1b664d99d0f93"
format_stamp: "Formatted at 2026-08-09 00:52:19 on dist-test-slave-j2vl"
server_key: "9d31299d5af461959360bc39ecaa8198"
server_key_iv: "7cc2edd2d161649cb04b0b0d71212da4"
server_key_version: "encryptionkey@0"
I20260809 00:52:19.565126 13421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "80d1702c61c443aaa1a1b664d99d0f93"
format_stamp: "Formatted at 2026-08-09 00:52:19 on dist-test-slave-j2vl"
server_key: "9d31299d5af461959360bc39ecaa8198"
server_key_iv: "7cc2edd2d161649cb04b0b0d71212da4"
server_key_version: "encryptionkey@0"
I20260809 00:52:19.571661 13421 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:52:19.576645 13446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:19.578527 13421 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:52:19.578783 13421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "80d1702c61c443aaa1a1b664d99d0f93"
format_stamp: "Formatted at 2026-08-09 00:52:19 on dist-test-slave-j2vl"
server_key: "9d31299d5af461959360bc39ecaa8198"
server_key_iv: "7cc2edd2d161649cb04b0b0d71212da4"
server_key_version: "encryptionkey@0"
I20260809 00:52:19.579037 13421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:19.604094 13421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:19.609050 13421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:19.609463 13421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:19.612447 13421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:19.612556 13421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:19.612725 13421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:19.612835 13421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 00:52:19 dist-test-slave-j2vl krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236739, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:19.641310 13463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:19.614161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44933 (local address 127.9.225.2:56765)
0809 00:52:19.615250 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:19.615327 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:19.618442 (+ 3115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:19.619495 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:19.619504 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:19.620718 (+ 1214us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:19.621874 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:19.621902 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:19.623684 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:19.623692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:19.625811 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:19.625841 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:19.626830 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:19.626839 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:19.627067 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:19.627871 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:19.627901 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:19.631026 (+ 3125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:19.635672 (+ 4646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:19.635683 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:19.635707 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:19.636498 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:19.637143 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:19.637148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:19.637153 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:19.637614 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:19.638418 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:19.638432 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:19.639105 (+ 673us) client_negotiation.cc:769] Sending connection context
0809 00:52:19.639574 (+ 469us) client_negotiation.cc:240] Negotiation successful
0809 00:52:19.640085 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":382,"threads_started":1}
I20260809 00:52:19.642383 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:19.613428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:56765 (local address 127.9.225.62:44933)
0809 00:52:19.613910 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:19.613917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:19.615488 (+ 1571us) server_negotiation.cc:407] Connection header received
0809 00:52:19.618933 (+ 3445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:19.618942 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:19.619054 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:19.619193 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:19.622112 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:19.623411 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:19.626079 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:19.626552 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:19.626695 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:19.631397 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:19.631407 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:19.631414 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:19.631496 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:19.635480 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:19.636786 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:19.636794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:19.636799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:19.637005 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:19.637774 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:19.637779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:19.637782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:19.638251 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:19.638440 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:19.641636 (+ 3196us) server_negotiation.cc:299] Negotiation successful
0809 00:52:19.641992 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":231,"threads_started":1}
I20260809 00:52:19.665063 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:19.649139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44933 (local address 127.9.225.2:38109)
0809 00:52:19.650131 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:19.650198 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:19.650393 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:19.650794 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:19.650799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:19.651330 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:19.651639 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:19.651649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:19.652688 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:19.652700 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:19.654140 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:19.654164 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:19.655227 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:19.655240 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:19.655429 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:19.656394 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:19.656413 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:19.657929 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:19.661847 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:19.661853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:19.661857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:19.662565 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:19.663009 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:19.663014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:19.663018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:19.663220 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:19.663907 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:19.663912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:19.664122 (+ 210us) client_negotiation.cc:769] Sending connection context
0809 00:52:19.664332 (+ 210us) client_negotiation.cc:240] Negotiation successful
0809 00:52:19.664704 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"thread_start_us":639,"threads_started":1}
I20260809 00:52:19.665557 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:19.649154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:38109 (local address 127.9.225.62:44933)
0809 00:52:19.649431 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:19.649440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:19.650301 (+ 861us) server_negotiation.cc:407] Connection header received
0809 00:52:19.650562 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:19.650566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:19.650642 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:19.650758 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:19.651795 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:19.652530 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:19.654329 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:19.655012 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:19.655165 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:19.658194 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:19.658200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:19.658203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:19.658241 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:19.661681 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:19.662735 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:19.662741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:19.662745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:19.662889 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:19.663389 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:19.663393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:19.663395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:19.663778 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:19.663960 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:19.664587 (+ 627us) server_negotiation.cc:299] Negotiation successful
0809 00:52:19.664988 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260809 00:52:19.674532 13421 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:39775
I20260809 00:52:19.674577 13621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:39775 every 8 connection(s)
I20260809 00:52:19.676200 13421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:52:19.679097 13623 heartbeater.cc:347] Connected to a master server at 127.9.225.62:44933
I20260809 00:52:19.679523 13623 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:19.680286 13623 heartbeater.cc:511] Master 127.9.225.62:44933 requested a full tablet report, sending...
I20260809 00:52:19.682026 13082 ts_manager.cc:194] Registered new tserver with Master: 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2:39775)
I20260809 00:52:19.682968 13082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:56765
I20260809 00:52:19.684082 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 13421
I20260809 00:52:19.684276 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:52:19.684897 10116 external_mini_cluster.cc:1428] Setting key b71b03b770de4bbfb94a9613c680abb2
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:52:19.742405 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44933
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:52:20.055480 13414 heartbeater.cc:503] Master 127.9.225.62:44933 was elected leader, sending a full tablet report...
W20260809 00:52:20.133448 13628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:20.133805 13628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:20.133924 13628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:20.141739 13628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:20.141891 13628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:20.141988 13628 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 00:52:20.142045 13628 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 00:52:20.142112 13628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:52:20.152859 13628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.225.62:44933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:20.154507 13628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:20.156008 13628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:20.171566 13637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:20.171490 13641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:20.171515 13643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:20.172677 13628 server_base.cc:1034] running on GCE node
I20260809 00:52:20.173666 13628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:20.174927 13628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:20.176230 13628 hybrid_clock.cc:648] HybridClock initialized: now 1786236740176124 us; error 66 us; skew 500 ppm
I20260809 00:52:20.176666 13628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:20 dist-test-slave-j2vl krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236740, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:20.180858 13628 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:52:20.182871 13628 webserver.cc:460] Webserver started at http://127.9.225.3:33747/ using document root <none> and password file <none>
I20260809 00:52:20.183826 13628 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:20.183996 13628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:20.184430 13628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:20.187953 13628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "4f7d14cf3d3e4b6a831203886c13359e"
format_stamp: "Formatted at 2026-08-09 00:52:20 on dist-test-slave-j2vl"
server_key: "3ccdafc716204d131f1e1b574c250639"
server_key_iv: "86a8797a9b4988896ce9a449933c49c5"
server_key_version: "encryptionkey@0"
I20260809 00:52:20.188885 13628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "4f7d14cf3d3e4b6a831203886c13359e"
format_stamp: "Formatted at 2026-08-09 00:52:20 on dist-test-slave-j2vl"
server_key: "3ccdafc716204d131f1e1b574c250639"
server_key_iv: "86a8797a9b4988896ce9a449933c49c5"
server_key_version: "encryptionkey@0"
I20260809 00:52:20.195613 13628 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260809 00:52:20.200133 13656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:20.201890 13628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:52:20.202147 13628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "4f7d14cf3d3e4b6a831203886c13359e"
format_stamp: "Formatted at 2026-08-09 00:52:20 on dist-test-slave-j2vl"
server_key: "3ccdafc716204d131f1e1b574c250639"
server_key_iv: "86a8797a9b4988896ce9a449933c49c5"
server_key_version: "encryptionkey@0"
I20260809 00:52:20.202412 13628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:20.223524 13628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:20.228791 13628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:20.229154 13628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:20.232436 13628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:20.232597 13628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:20.232738 13628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:20.232892 13628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 09 00:52:20 dist-test-slave-j2vl krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236740, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:20.263239 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.233085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:43259 (local address 127.9.225.62:44933)
0809 00:52:20.234131 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:20.234135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:20.235175 (+ 1040us) server_negotiation.cc:407] Connection header received
0809 00:52:20.237550 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:20.237556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:20.237666 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:20.237803 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:20.240647 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.242156 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.246078 (+ 3922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.246871 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.247060 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.253024 (+ 5964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:20.253038 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:20.253046 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:20.253117 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:20.257992 (+ 4875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:20.259492 (+ 1500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:20.259501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:20.259507 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:20.259734 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:20.260360 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:20.260367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:20.260375 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:20.260701 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:20.260918 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:20.262281 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0809 00:52:20.262678 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":787,"threads_started":1}
I20260809 00:52:20.264781 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.234004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44933 (local address 127.9.225.3:43259)
0809 00:52:20.234969 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:20.235025 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:20.237119 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:20.238112 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:20.238122 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:20.239090 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:20.240261 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.240291 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.242497 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.242507 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.245814 (+ 3307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.245842 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.247291 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.247302 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.247607 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.248720 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:20.248751 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:20.252624 (+ 3873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:20.258255 (+ 5631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:20.258271 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:20.258300 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:20.259299 (+ 999us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:20.259915 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:20.259922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:20.259935 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:20.260210 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:20.260899 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:20.260909 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:20.261578 (+ 669us) client_negotiation.cc:769] Sending connection context
0809 00:52:20.262097 (+ 519us) client_negotiation.cc:240] Negotiation successful
0809 00:52:20.263001 (+ 904us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":435,"threads_started":1}
I20260809 00:52:20.293021 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.273128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:33975 (local address 127.9.225.62:44933)
0809 00:52:20.273410 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:20.273419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:20.274253 (+ 834us) server_negotiation.cc:407] Connection header received
0809 00:52:20.274429 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:20.274438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:20.274583 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:20.274758 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:20.276314 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.277930 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.281037 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.281942 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.282087 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.285615 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:20.285625 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:20.285633 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:20.285690 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:20.289160 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:20.290545 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:20.290551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:20.290554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:20.290767 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:20.291295 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:20.291300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:20.291305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:20.291646 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:20.291842 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:20.292329 (+ 487us) server_negotiation.cc:299] Negotiation successful
0809 00:52:20.292618 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 00:52:20.293102 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.273094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44933 (local address 127.9.225.3:33975)
0809 00:52:20.274111 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:20.274129 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:20.274280 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:20.274783 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:20.274786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:20.275565 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:20.276079 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.276098 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.278265 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.278273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.280710 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.280743 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.282515 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.282524 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.282697 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.283549 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:20.283574 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:20.285223 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:20.289402 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:20.289412 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:20.289419 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:20.290317 (+ 898us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:20.290942 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:20.290948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:20.290951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:20.291118 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:20.291800 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:20.291805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:20.292029 (+ 224us) client_negotiation.cc:769] Sending connection context
0809 00:52:20.292192 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 00:52:20.292618 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":609,"threads_started":1}
I20260809 00:52:20.299737 13628 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:42447
I20260809 00:52:20.299738 13839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:42447 every 8 connection(s)
I20260809 00:52:20.301308 13628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:52:20.303797 13840 heartbeater.cc:347] Connected to a master server at 127.9.225.62:44933
I20260809 00:52:20.304111 13840 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:20.304924 13840 heartbeater.cc:511] Master 127.9.225.62:44933 requested a full tablet report, sending...
I20260809 00:52:20.307039 13082 ts_manager.cc:194] Registered new tserver with Master: 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3:42447)
I20260809 00:52:20.307396 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 13628
I20260809 00:52:20.307587 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:52:20.308153 10116 external_mini_cluster.cc:1428] Setting key 16e785ed3c0a67393534317d660f2c13
I20260809 00:52:20.308156 13082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:43259
I20260809 00:52:20.311726 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 00:52:20 dist-test-slave-j2vl krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 00:52:20 dist-test-slave-j2vl krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:20.382138 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.358024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.9.225.62:44933)
0809 00:52:20.358233 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:20.358237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:20.358713 (+ 476us) server_negotiation.cc:407] Connection header received
0809 00:52:20.358899 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:20.358904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:20.359014 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:20.359120 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:20.360853 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.361758 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.363139 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.363766 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.363896 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.376046 (+ 12150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:20.376053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:20.376056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:20.376116 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:20.379172 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:20.380060 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:20.380065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:20.380068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:20.380228 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:20.380687 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:20.380691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:20.380694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:20.381025 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:20.381257 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:20.381635 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 00:52:20.381855 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 00:52:20.387200 13082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:52:20.394841 13082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:52:20.439014 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.411224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.9.225.2:39775)
0809 00:52:20.426503 (+ 15279us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:20.426514 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:20.428258 (+ 1744us) server_negotiation.cc:407] Connection header received
0809 00:52:20.428388 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:20.428397 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:20.428704 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:20.428920 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:20.430320 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.431939 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.435651 (+ 3712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.437008 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.437149 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.438368 (+ 1219us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:20.438471 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 00:52:20.438670 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15082,"thread_start_us":15289,"threads_started":1}
I20260809 00:52:20.439116 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.410209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:32995 (local address 127.0.0.1:42840)
0809 00:52:20.426230 (+ 16021us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:20.426307 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:20.426553 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:20.429197 (+ 2644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:20.429207 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:20.429249 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:20.430064 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.430078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.432119 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.432125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.435908 (+ 3783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.435926 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.438202 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.438208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.438414 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.438450 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 00:52:20.438540 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 00:52:20.438689 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15593,"mutex_wait_us":898,"spinlock_wait_cycles":896,"thread_start_us":16551,"threads_started":1}
I20260809 00:52:20.438719 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.410614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:39775 (local address 127.0.0.1:56972)
0809 00:52:20.427676 (+ 17062us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:20.427693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:20.427864 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:20.429197 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:20.429207 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:20.429254 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:20.430064 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.430078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.432119 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.432125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.435426 (+ 3301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.435439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.437524 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.437531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.437771 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.437832 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 00:52:20.437967 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 00:52:20.438202 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16953,"mutex_wait_us":242,"spinlock_wait_cycles":3456,"thread_start_us":16848,"threads_started":1}
I20260809 00:52:20.439961 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.410422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.9.225.1:32995)
0809 00:52:20.428408 (+ 17986us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:20.428417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:20.428442 (+ 25us) server_negotiation.cc:407] Connection header received
0809 00:52:20.428525 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:20.428531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:20.428705 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:20.428920 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:20.430320 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.431944 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.436130 (+ 4186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.437683 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.437817 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.438054 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:20.438848 (+ 794us) server_negotiation.cc:299] Negotiation successful
0809 00:52:20.439039 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17810,"thread_start_us":17656,"threads_started":1}
I20260809 00:52:20.443113 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.411226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:42447 (local address 127.0.0.1:33392)
0809 00:52:20.435082 (+ 23856us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:20.435104 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:20.435273 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:20.436132 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:20.436139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:20.436170 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:20.436574 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.436589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.439034 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.439046 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.440729 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.440745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.442248 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.442255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.442458 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.442505 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 00:52:20.442610 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 00:52:20.442759 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23669,"thread_start_us":22703,"threads_started":1}
I20260809 00:52:20.443203 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.411543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.9.225.3:42447)
0809 00:52:20.435013 (+ 23470us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:20.435021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:20.435253 (+ 232us) server_negotiation.cc:407] Connection header received
0809 00:52:20.435524 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:20.435536 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:20.435861 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:20.436149 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:20.437146 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.438835 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.440917 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.442026 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.442191 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.442449 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:20.442679 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 00:52:20.442849 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23279,"thread_start_us":23016,"threads_started":1}
I20260809 00:52:20.443894 13312 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a (DEFAULT_TABLE table=test-table [id=28f9da2cd19b4b909e70bed513834d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:20.445842 13312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:20.445567 13527 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a (DEFAULT_TABLE table=test-table [id=28f9da2cd19b4b909e70bed513834d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:20.447237 13527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:20.447724 13734 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a (DEFAULT_TABLE table=test-table [id=28f9da2cd19b4b909e70bed513834d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:20.449437 13734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:20.470700 13870 tablet_bootstrap.cc:492] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced: Bootstrap starting.
I20260809 00:52:20.472131 13868 tablet_bootstrap.cc:492] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e: Bootstrap starting.
I20260809 00:52:20.473197 13874 tablet_bootstrap.cc:492] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93: Bootstrap starting.
I20260809 00:52:20.477236 13868 tablet_bootstrap.cc:654] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:20.478974 13868 log.cc:824] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:20.479559 13874 tablet_bootstrap.cc:654] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:20.481482 13874 log.cc:824] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:20.481747 13870 tablet_bootstrap.cc:654] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:20.483522 13870 log.cc:824] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:20.487067 13874 tablet_bootstrap.cc:492] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93: No bootstrap required, opened a new log
I20260809 00:52:20.487543 13874 ts_tablet_manager.cc:1397] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.002s
I20260809 00:52:20.487753 13868 tablet_bootstrap.cc:492] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e: No bootstrap required, opened a new log
I20260809 00:52:20.488102 13868 ts_tablet_manager.cc:1397] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20260809 00:52:20.500754 13874 raft_consensus.cc:348] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.501473 13874 raft_consensus.cc:374] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:20.501621 13874 raft_consensus.cc:729] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d1702c61c443aaa1a1b664d99d0f93, State: Initialized, Role: FOLLOWER
I20260809 00:52:20.502591 13874 consensus_queue.cc:260] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.504182 13623 heartbeater.cc:503] Master 127.9.225.62:44933 was elected leader, sending a full tablet report...
I20260809 00:52:20.505270 13874 ts_tablet_manager.cc:1428] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93: Time spent starting tablet: real 0.017s user 0.014s sys 0.002s
I20260809 00:52:20.505489 13870 tablet_bootstrap.cc:492] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced: No bootstrap required, opened a new log
I20260809 00:52:20.506013 13870 ts_tablet_manager.cc:1397] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.010s
I20260809 00:52:20.505865 13868 raft_consensus.cc:348] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.506596 13868 raft_consensus.cc:374] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:20.506808 13868 raft_consensus.cc:729] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d14cf3d3e4b6a831203886c13359e, State: Initialized, Role: FOLLOWER
I20260809 00:52:20.508023 13868 consensus_queue.cc:260] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.510296 13840 heartbeater.cc:503] Master 127.9.225.62:44933 was elected leader, sending a full tablet report...
I20260809 00:52:20.511461 13868 ts_tablet_manager.cc:1428] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e: Time spent starting tablet: real 0.023s user 0.020s sys 0.003s
I20260809 00:52:20.520912 13870 raft_consensus.cc:348] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.521596 13870 raft_consensus.cc:374] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:20.521708 13870 raft_consensus.cc:729] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1a6f3c288b04133af66a94d252e8ced, State: Initialized, Role: FOLLOWER
I20260809 00:52:20.522581 13870 consensus_queue.cc:260] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.527565 13870 ts_tablet_manager.cc:1428] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced: Time spent starting tablet: real 0.021s user 0.013s sys 0.005s
W20260809 00:52:20.547108 13415 tablet.cc:2369] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:52:20.553398 13841 tablet.cc:2369] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:52:20.679109 13624 tablet.cc:2369] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:52:20.719071 13881 raft_consensus.cc:482] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:52:20.719455 13881 raft_consensus.cc:504] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.721410 13881 leader_election.cc:288] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995), 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2:39775)
I20260809 00:52:20.728402 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.721899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:39775 (local address 127.9.225.3:35847)
0809 00:52:20.722242 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:20.722265 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:20.722470 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:20.722975 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:20.722981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:20.723012 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:20.723260 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.723276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.724802 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.724806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.726418 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.726431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.727581 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.727587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.727779 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.727805 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 00:52:20.727925 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 00:52:20.728086 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260809 00:52:20.728395 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.721961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:35847 (local address 127.9.225.2:39775)
0809 00:52:20.722158 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:20.722166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:20.722384 (+ 218us) server_negotiation.cc:407] Connection header received
0809 00:52:20.722674 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:20.722684 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:20.722822 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:20.722931 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:20.723583 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.724653 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.726598 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.727391 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.727556 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.727671 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:20.727955 (+ 284us) server_negotiation.cc:299] Negotiation successful
0809 00:52:20.728101 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 00:52:20.728845 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.721884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:54491 (local address 127.9.225.1:32995)
0809 00:52:20.722119 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:20.722125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:20.723517 (+ 1392us) server_negotiation.cc:407] Connection header received
0809 00:52:20.723606 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:20.723611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:20.723715 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:20.723821 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:20.724376 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.725233 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.727003 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.727805 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.727981 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.728101 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:20.728432 (+ 331us) server_negotiation.cc:299] Negotiation successful
0809 00:52:20.728571 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 00:52:20.728888 13883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.722343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:32995 (local address 127.9.225.3:54491)
0809 00:52:20.723179 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:20.723201 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:20.723332 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:20.723841 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:20.723844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:20.723868 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:20.724192 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.724199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.725431 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.725438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.726822 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.726835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.728069 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.728075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.728240 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.728266 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 00:52:20.728392 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 00:52:20.728531 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":503,"threads_started":1}
I20260809 00:52:20.729636 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4ed1d6fe5c463c9d2ca045a5374a4a" candidate_uuid: "4f7d14cf3d3e4b6a831203886c13359e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d1702c61c443aaa1a1b664d99d0f93" is_pre_election: true
I20260809 00:52:20.729883 13347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4ed1d6fe5c463c9d2ca045a5374a4a" candidate_uuid: "4f7d14cf3d3e4b6a831203886c13359e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1a6f3c288b04133af66a94d252e8ced" is_pre_election: true
I20260809 00:52:20.730244 13552 raft_consensus.cc:2454] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7d14cf3d3e4b6a831203886c13359e in term 0.
I20260809 00:52:20.730473 13347 raft_consensus.cc:2454] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7d14cf3d3e4b6a831203886c13359e in term 0.
I20260809 00:52:20.731029 13657 leader_election.cc:302] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7d14cf3d3e4b6a831203886c13359e, 80d1702c61c443aaa1a1b664d99d0f93; no voters:
I20260809 00:52:20.731477 13881 raft_consensus.cc:2781] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:52:20.731648 13881 raft_consensus.cc:482] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:52:20.731768 13881 raft_consensus.cc:3037] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:20.734180 13881 raft_consensus.cc:504] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.734874 13881 leader_election.cc:288] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [CANDIDATE]: Term 1 election: Requested vote from peers c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995), 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2:39775)
I20260809 00:52:20.735383 13347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4ed1d6fe5c463c9d2ca045a5374a4a" candidate_uuid: "4f7d14cf3d3e4b6a831203886c13359e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1a6f3c288b04133af66a94d252e8ced"
I20260809 00:52:20.735445 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4ed1d6fe5c463c9d2ca045a5374a4a" candidate_uuid: "4f7d14cf3d3e4b6a831203886c13359e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d1702c61c443aaa1a1b664d99d0f93"
I20260809 00:52:20.735622 13347 raft_consensus.cc:3037] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:20.735662 13552 raft_consensus.cc:3037] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:20.738117 13347 raft_consensus.cc:2454] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7d14cf3d3e4b6a831203886c13359e in term 1.
I20260809 00:52:20.738103 13552 raft_consensus.cc:2454] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7d14cf3d3e4b6a831203886c13359e in term 1.
I20260809 00:52:20.738740 13663 leader_election.cc:302] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7d14cf3d3e4b6a831203886c13359e, c1a6f3c288b04133af66a94d252e8ced; no voters:
I20260809 00:52:20.739161 13881 raft_consensus.cc:2781] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:20.739720 13881 raft_consensus.cc:686] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 1 LEADER]: Becoming Leader. State: Replica: 4f7d14cf3d3e4b6a831203886c13359e, State: Running, Role: LEADER
I20260809 00:52:20.740276 13881 consensus_queue.cc:237] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.745869 13081 catalog_manager.cc:5495] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3). New cstate: current_term: 1 leader_uuid: "4f7d14cf3d3e4b6a831203886c13359e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } health_report { overall_health: HEALTHY } } }
Aug 09 00:52:20 dist-test-slave-j2vl krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236738, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:20.857827 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.840428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44933 (local address 127.0.0.1:60710)
0809 00:52:20.841090 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:20.841108 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:20.841248 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:20.841647 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:20.841650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:20.842434 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:20.842734 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.842745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.844003 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.844008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.845944 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.845971 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.847310 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.847322 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.847477 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.848309 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:20.848336 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:20.851104 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:20.854806 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:20.854827 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:20.854836 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:20.855480 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:20.855930 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:20.855934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:20.855936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:20.856089 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:20.856697 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:20.856704 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:20.856926 (+ 222us) client_negotiation.cc:769] Sending connection context
0809 00:52:20.857093 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 00:52:20.857422 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":371,"threads_started":1}
I20260809 00:52:20.857919 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.840347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.9.225.62:44933)
0809 00:52:20.840585 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:20.840593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:20.841207 (+ 614us) server_negotiation.cc:407] Connection header received
0809 00:52:20.841420 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:20.841427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:20.841501 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:20.841586 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:20.842893 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.843702 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.846187 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.846744 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.846880 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.851427 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:20.851435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:20.851438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:20.851483 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:20.854596 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:20.855636 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:20.855642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:20.855644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:20.855797 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:20.856214 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:20.856219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:20.856221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:20.856544 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:20.856667 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:20.857220 (+ 553us) server_negotiation.cc:299] Negotiation successful
0809 00:52:20.857608 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":83}
I20260809 00:52:20.864565 13081 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:60710:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 00:52:20.865989 13081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:52:20.874558 13312 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8a59c05c6d47439ffb853cf14793ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bba659c741564928b74229724208b7f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 00:52:20.874761 13527 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8a59c05c6d47439ffb853cf14793ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bba659c741564928b74229724208b7f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 00:52:20.875845 13312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8a59c05c6d47439ffb853cf14793ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:20.875847 13527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8a59c05c6d47439ffb853cf14793ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:20.875365 13734 tablet_service.cc:1467] Processing CreateTablet for tablet 1b8a59c05c6d47439ffb853cf14793ff (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bba659c741564928b74229724208b7f5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 00:52:20.876274 13734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b8a59c05c6d47439ffb853cf14793ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:20.881924 13870 tablet_bootstrap.cc:492] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced: Bootstrap starting.
I20260809 00:52:20.884063 13874 tablet_bootstrap.cc:492] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93: Bootstrap starting.
I20260809 00:52:20.884182 13870 tablet_bootstrap.cc:654] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:20.885021 13868 tablet_bootstrap.cc:492] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: Bootstrap starting.
I20260809 00:52:20.885957 13874 tablet_bootstrap.cc:654] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:20.886722 13868 tablet_bootstrap.cc:654] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:20.887074 13870 tablet_bootstrap.cc:492] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced: No bootstrap required, opened a new log
I20260809 00:52:20.887233 13870 ts_tablet_manager.cc:1397] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260809 00:52:20.888219 13870 raft_consensus.cc:348] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.888566 13870 raft_consensus.cc:374] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:20.888682 13870 raft_consensus.cc:729] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1a6f3c288b04133af66a94d252e8ced, State: Initialized, Role: FOLLOWER
I20260809 00:52:20.888813 13874 tablet_bootstrap.cc:492] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93: No bootstrap required, opened a new log
I20260809 00:52:20.888998 13874 ts_tablet_manager.cc:1397] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 00:52:20.889029 13870 consensus_queue.cc:260] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.889930 13874 raft_consensus.cc:348] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.889722 13882 tablet_replica.cc:439] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } } }
I20260809 00:52:20.890203 13874 raft_consensus.cc:374] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:20.890240 13882 tablet_replica.cc:442] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:20.890306 13874 raft_consensus.cc:729] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d1702c61c443aaa1a1b664d99d0f93, State: Initialized, Role: FOLLOWER
I20260809 00:52:20.890573 13874 consensus_queue.cc:260] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.890760 13870 ts_tablet_manager.cc:1428] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260809 00:52:20.891131 13880 tablet_replica.cc:439] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } } }
I20260809 00:52:20.891512 13880 tablet_replica.cc:442] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:20.892679 13868 tablet_bootstrap.cc:492] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: No bootstrap required, opened a new log
I20260809 00:52:20.892895 13868 ts_tablet_manager.cc:1397] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260809 00:52:20.894045 13868 raft_consensus.cc:348] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.894330 13868 raft_consensus.cc:374] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:20.894464 13868 raft_consensus.cc:729] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d14cf3d3e4b6a831203886c13359e, State: Initialized, Role: FOLLOWER
I20260809 00:52:20.894523 13874 ts_tablet_manager.cc:1428] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260809 00:52:20.894812 13868 consensus_queue.cc:260] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.895596 13868 ts_tablet_manager.cc:1428] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 00:52:20.895390 13881 tablet_replica.cc:439] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } } }
I20260809 00:52:20.896354 13881 tablet_replica.cc:442] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:20.925498 13881 raft_consensus.cc:482] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:52:20.925861 13881 raft_consensus.cc:504] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.926842 13881 leader_election.cc:288] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995), 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2:39775)
I20260809 00:52:20.927421 13347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8a59c05c6d47439ffb853cf14793ff" candidate_uuid: "4f7d14cf3d3e4b6a831203886c13359e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1a6f3c288b04133af66a94d252e8ced" is_pre_election: true
I20260809 00:52:20.927575 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8a59c05c6d47439ffb853cf14793ff" candidate_uuid: "4f7d14cf3d3e4b6a831203886c13359e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d1702c61c443aaa1a1b664d99d0f93" is_pre_election: true
I20260809 00:52:20.927755 13347 raft_consensus.cc:2454] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7d14cf3d3e4b6a831203886c13359e in term 0.
I20260809 00:52:20.927883 13552 raft_consensus.cc:2454] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7d14cf3d3e4b6a831203886c13359e in term 0.
I20260809 00:52:20.928293 13663 leader_election.cc:302] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7d14cf3d3e4b6a831203886c13359e, c1a6f3c288b04133af66a94d252e8ced; no voters:
I20260809 00:52:20.928625 13881 raft_consensus.cc:2781] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:52:20.928745 13881 raft_consensus.cc:482] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:52:20.928885 13881 raft_consensus.cc:3037] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:20.931218 13881 raft_consensus.cc:504] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.931998 13881 leader_election.cc:288] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [CANDIDATE]: Term 1 election: Requested vote from peers c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995), 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2:39775)
I20260809 00:52:20.932550 13347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8a59c05c6d47439ffb853cf14793ff" candidate_uuid: "4f7d14cf3d3e4b6a831203886c13359e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1a6f3c288b04133af66a94d252e8ced"
I20260809 00:52:20.932570 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b8a59c05c6d47439ffb853cf14793ff" candidate_uuid: "4f7d14cf3d3e4b6a831203886c13359e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d1702c61c443aaa1a1b664d99d0f93"
I20260809 00:52:20.932806 13347 raft_consensus.cc:3037] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:20.932833 13552 raft_consensus.cc:3037] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:20.935041 13347 raft_consensus.cc:2454] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7d14cf3d3e4b6a831203886c13359e in term 1.
I20260809 00:52:20.935055 13552 raft_consensus.cc:2454] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7d14cf3d3e4b6a831203886c13359e in term 1.
I20260809 00:52:20.935590 13657 leader_election.cc:302] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7d14cf3d3e4b6a831203886c13359e, 80d1702c61c443aaa1a1b664d99d0f93; no voters:
I20260809 00:52:20.936033 13881 raft_consensus.cc:2781] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:20.936269 13881 raft_consensus.cc:686] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [term 1 LEADER]: Becoming Leader. State: Replica: 4f7d14cf3d3e4b6a831203886c13359e, State: Running, Role: LEADER
I20260809 00:52:20.936535 13881 consensus_queue.cc:237] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:20.938411 13884 tablet_replica.cc:439] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: TxnStatusTablet state changed. Reason: New leader 4f7d14cf3d3e4b6a831203886c13359e. Latest consensus state: current_term: 1 leader_uuid: "4f7d14cf3d3e4b6a831203886c13359e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } } }
I20260809 00:52:20.938778 13884 tablet_replica.cc:442] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: This TxnStatusTablet replica's current role is: LEADER
I20260809 00:52:20.939607 13899 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 00:52:20.939980 13899 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 00:52:20.940841 13078 catalog_manager.cc:5495] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3). New cstate: current_term: 1 leader_uuid: "4f7d14cf3d3e4b6a831203886c13359e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } health_report { overall_health: HEALTHY } } }
I20260809 00:52:20.968812 13892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.960983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:42447 (local address 127.0.0.1:33408)
0809 00:52:20.961283 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:20.961309 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:20.961543 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:20.962177 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:20.962181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:20.962192 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 00:52:20.962369 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.962377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.964030 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.964035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.965239 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:20.965254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.966404 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.966414 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:20.966761 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.966860 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 00:52:20.968197 (+ 1337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 00:52:20.968222 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 00:52:20.968308 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 00:52:20.968468 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 00:52:20.968849 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:20.961077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.9.225.3:42447)
0809 00:52:20.961667 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:20.961674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:20.961695 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:52:20.961801 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:20.961807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:20.961936 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:20.962117 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:20.962563 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.963857 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.965435 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:20.966153 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:20.966327 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:20.967148 (+ 821us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:20.968041 (+ 893us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:20.968218 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:20.968379 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 00:52:20.968481 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":303,"threads_started":1}
I20260809 00:52:20.978642 13347 raft_consensus.cc:1261] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Refusing update from remote peer 4f7d14cf3d3e4b6a831203886c13359e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:20.978799 13552 raft_consensus.cc:1261] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Refusing update from remote peer 4f7d14cf3d3e4b6a831203886c13359e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:20.979962 13884 consensus_queue.cc:1035] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:20.981190 13884 consensus_queue.cc:1035] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:20.994045 13880 tablet_replica.cc:439] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93: TxnStatusTablet state changed. Reason: New leader 4f7d14cf3d3e4b6a831203886c13359e. Latest consensus state: current_term: 1 leader_uuid: "4f7d14cf3d3e4b6a831203886c13359e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } } }
I20260809 00:52:20.994515 13880 tablet_replica.cc:442] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:21.002704 13884 tablet_replica.cc:439] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f7d14cf3d3e4b6a831203886c13359e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } } }
I20260809 00:52:21.003100 13884 tablet_replica.cc:442] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: This TxnStatusTablet replica's current role is: LEADER
I20260809 00:52:21.004805 13904 mvcc.cc:205] Tried to move back new op lower bound from 7316425691034157056 to 7316425690878627840. Current Snapshot: MvccSnapshot[applied={T|T < 7316425691034157056}]
I20260809 00:52:21.016589 13880 tablet_replica.cc:439] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f7d14cf3d3e4b6a831203886c13359e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } } }
I20260809 00:52:21.017047 13880 tablet_replica.cc:442] T 1b8a59c05c6d47439ffb853cf14793ff P 80d1702c61c443aaa1a1b664d99d0f93: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:21.022924 13882 tablet_replica.cc:439] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced: TxnStatusTablet state changed. Reason: New leader 4f7d14cf3d3e4b6a831203886c13359e. Latest consensus state: current_term: 1 leader_uuid: "4f7d14cf3d3e4b6a831203886c13359e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } } }
I20260809 00:52:21.023502 13882 tablet_replica.cc:442] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:21.030166 13906 mvcc.cc:205] Tried to move back new op lower bound from 7316425691034157056 to 7316425690878627840. Current Snapshot: MvccSnapshot[applied={T|T < 7316425691034157056}]
I20260809 00:52:21.030412 13884 tablet_replica.cc:439] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f7d14cf3d3e4b6a831203886c13359e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } } }
I20260809 00:52:21.030795 13884 tablet_replica.cc:442] T 1b8a59c05c6d47439ffb853cf14793ff P 4f7d14cf3d3e4b6a831203886c13359e: This TxnStatusTablet replica's current role is: LEADER
I20260809 00:52:21.033077 13882 tablet_replica.cc:439] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f7d14cf3d3e4b6a831203886c13359e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } } }
I20260809 00:52:21.033428 13882 tablet_replica.cc:442] T 1b8a59c05c6d47439ffb853cf14793ff P c1a6f3c288b04133af66a94d252e8ced: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:52:21.044689 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:21.032350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.9.225.3:42447)
0809 00:52:21.032589 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:21.032596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:21.038358 (+ 5762us) server_negotiation.cc:407] Connection header received
0809 00:52:21.038449 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:21.038455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:21.038563 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:21.038671 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:21.039132 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.040327 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.042361 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.043057 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.043214 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:21.043748 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:21.043911 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:21.044008 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:21.044247 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 00:52:21.044345 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 00:52:21.069270 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:21.061622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:33345 (local address 127.9.225.3:42447)
0809 00:52:21.061798 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:21.061807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:21.062351 (+ 544us) server_negotiation.cc:407] Connection header received
0809 00:52:21.062621 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:21.062626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:21.062770 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:21.062874 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:21.063576 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.064993 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.066781 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.067515 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.067665 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:21.068288 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:21.068483 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:21.068582 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:21.068797 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 00:52:21.068890 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 00:52:21.069640 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:21.060817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:42447 (local address 127.9.225.3:33345)
0809 00:52:21.062207 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:21.062234 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:21.062447 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:21.062931 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:21.062937 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:21.062957 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 00:52:21.063327 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:21.063340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.065153 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.065159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:21.066590 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:21.066614 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.067823 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.067830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:21.067972 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:21.068050 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 00:52:21.068631 (+ 581us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 00:52:21.068645 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 00:52:21.068772 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 00:52:21.068915 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"thread_start_us":984,"threads_started":1}
I20260809 00:52:21.081964 13552 raft_consensus.cc:1261] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Refusing update from remote peer 4f7d14cf3d3e4b6a831203886c13359e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:21.082115 13345 raft_consensus.cc:1261] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Refusing update from remote peer 4f7d14cf3d3e4b6a831203886c13359e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:21.082940 13884 consensus_queue.cc:1035] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:21.083376 13881 consensus_queue.cc:1035] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:21.169188 13082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:21.169590 13082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:21.173693 13082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866: Sending DeleteTablet for 3 replicas of tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a
I20260809 00:52:21.175153 13312 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:21 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:42840
I20260809 00:52:21.175177 13732 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:21 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:33392
I20260809 00:52:21.175222 13527 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:21 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:56972
I20260809 00:52:21.178836 13078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:52:21.179873 13078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:52:21.185007 13931 tablet_replica.cc:330] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93: stopping tablet replica
I20260809 00:52:21.185734 13931 raft_consensus.cc:2229] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:21.186332 13931 raft_consensus.cc:2258] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:21.188804 13931 ts_tablet_manager.cc:1905] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:21.189592 13932 tablet_replica.cc:330] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e: stopping tablet replica
I20260809 00:52:21.190202 13932 raft_consensus.cc:2229] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:52:21.191805 13311 tablet_service.cc:1467] Processing CreateTablet for tablet 70373523fc9f4c87ab2d9098c24b6251 (DEFAULT_TABLE table=test-table [id=6af3df203cd946e1a51d7ebaa1d00a74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:21.192611 13311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70373523fc9f4c87ab2d9098c24b6251. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:21.197762 13930 tablet_replica.cc:330] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced: stopping tablet replica
I20260809 00:52:21.197890 13527 tablet_service.cc:1467] Processing CreateTablet for tablet 70373523fc9f4c87ab2d9098c24b6251 (DEFAULT_TABLE table=test-table [id=6af3df203cd946e1a51d7ebaa1d00a74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:21.198462 13527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70373523fc9f4c87ab2d9098c24b6251. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:21.198511 13930 raft_consensus.cc:2229] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:21.198956 13930 raft_consensus.cc:2258] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:21.200433 13932 raft_consensus.cc:2258] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:21.201315 13931 ts_tablet_manager.cc:1918] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:52:21.201649 13931 log.cc:1196] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/0e4ed1d6fe5c463c9d2ca045a5374a4a
I20260809 00:52:21.202396 13931 ts_tablet_manager.cc:1939] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 80d1702c61c443aaa1a1b664d99d0f93: Deleting consensus metadata
I20260809 00:52:21.203227 13932 ts_tablet_manager.cc:1905] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:21.206158 13046 catalog_manager.cc:4841] TS 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2:39775): tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a (table test-table [id=28f9da2cd19b4b909e70bed513834d99]) successfully deleted
I20260809 00:52:21.208206 13870 tablet_bootstrap.cc:492] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced: Bootstrap starting.
I20260809 00:52:21.210207 13870 tablet_bootstrap.cc:654] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:21.210932 13874 tablet_bootstrap.cc:492] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93: Bootstrap starting.
I20260809 00:52:21.211853 13732 tablet_service.cc:1467] Processing CreateTablet for tablet 70373523fc9f4c87ab2d9098c24b6251 (DEFAULT_TABLE table=test-table [id=6af3df203cd946e1a51d7ebaa1d00a74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:21.212432 13732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70373523fc9f4c87ab2d9098c24b6251. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:21.212921 13874 tablet_bootstrap.cc:654] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:21.214335 13932 ts_tablet_manager.cc:1918] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:52:21.214643 13930 ts_tablet_manager.cc:1905] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:21.215329 13870 tablet_bootstrap.cc:492] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced: No bootstrap required, opened a new log
I20260809 00:52:21.215529 13870 ts_tablet_manager.cc:1397] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260809 00:52:21.216405 13870 raft_consensus.cc:348] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:21.216701 13870 raft_consensus.cc:374] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:21.216815 13870 raft_consensus.cc:729] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1a6f3c288b04133af66a94d252e8ced, State: Initialized, Role: FOLLOWER
I20260809 00:52:21.217087 13870 consensus_queue.cc:260] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:21.217756 13870 ts_tablet_manager.cc:1428] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 00:52:21.221092 13874 tablet_bootstrap.cc:492] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93: No bootstrap required, opened a new log
I20260809 00:52:21.221408 13874 ts_tablet_manager.cc:1397] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20260809 00:52:21.222492 13874 raft_consensus.cc:348] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:21.222694 13932 log.cc:1196] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/0e4ed1d6fe5c463c9d2ca045a5374a4a
I20260809 00:52:21.222988 13868 tablet_bootstrap.cc:492] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e: Bootstrap starting.
I20260809 00:52:21.223030 13874 raft_consensus.cc:374] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:21.223229 13874 raft_consensus.cc:729] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d1702c61c443aaa1a1b664d99d0f93, State: Initialized, Role: FOLLOWER
I20260809 00:52:21.223433 13932 ts_tablet_manager.cc:1939] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P 4f7d14cf3d3e4b6a831203886c13359e: Deleting consensus metadata
I20260809 00:52:21.223594 13874 consensus_queue.cc:260] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:21.224265 13874 ts_tablet_manager.cc:1428] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260809 00:52:21.225085 13868 tablet_bootstrap.cc:654] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:21.226104 13048 catalog_manager.cc:4841] TS 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3:42447): tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a (table test-table [id=28f9da2cd19b4b909e70bed513834d99]) successfully deleted
I20260809 00:52:21.231213 13930 ts_tablet_manager.cc:1918] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:52:21.231451 13930 log.cc:1196] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/0e4ed1d6fe5c463c9d2ca045a5374a4a
I20260809 00:52:21.231945 13930 ts_tablet_manager.cc:1939] T 0e4ed1d6fe5c463c9d2ca045a5374a4a P c1a6f3c288b04133af66a94d252e8ced: Deleting consensus metadata
I20260809 00:52:21.234064 13049 catalog_manager.cc:4841] TS c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995): tablet 0e4ed1d6fe5c463c9d2ca045a5374a4a (table test-table [id=28f9da2cd19b4b909e70bed513834d99]) successfully deleted
I20260809 00:52:21.235891 13868 tablet_bootstrap.cc:492] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e: No bootstrap required, opened a new log
I20260809 00:52:21.236129 13868 ts_tablet_manager.cc:1397] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260809 00:52:21.237110 13868 raft_consensus.cc:348] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:21.237471 13868 raft_consensus.cc:374] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:21.237577 13868 raft_consensus.cc:729] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d14cf3d3e4b6a831203886c13359e, State: Initialized, Role: FOLLOWER
I20260809 00:52:21.237896 13868 consensus_queue.cc:260] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:21.238829 13868 ts_tablet_manager.cc:1428] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 00:52:21.486610 13880 raft_consensus.cc:482] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:52:21.486922 13880 raft_consensus.cc:504] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:21.489147 13880 leader_election.cc:288] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995), 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3:42447)
I20260809 00:52:21.508538 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:21.489504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:32995 (local address 127.9.225.2:40377)
0809 00:52:21.496797 (+ 7293us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:21.496833 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:21.497070 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:21.498654 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:21.498661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:21.498694 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:21.499544 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:21.499558 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.501919 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.501924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:21.504279 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:21.504300 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.506937 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.506945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:21.507193 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:21.507236 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 00:52:21.507562 (+ 326us) client_negotiation.cc:240] Negotiation successful
0809 00:52:21.507789 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7147,"thread_start_us":6992,"threads_started":1}
I20260809 00:52:21.509812 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:21.489655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:40377 (local address 127.9.225.1:32995)
0809 00:52:21.498202 (+ 8547us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:21.498209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:21.498278 (+ 69us) server_negotiation.cc:407] Connection header received
0809 00:52:21.498360 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:21.498367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:21.498479 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:21.499841 (+ 1362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:21.500086 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.501750 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.504570 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.506631 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.506887 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:21.507038 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:21.508915 (+ 1877us) server_negotiation.cc:299] Negotiation successful
0809 00:52:21.509186 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8410,"thread_start_us":8069,"threads_started":1}
I20260809 00:52:21.510567 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:21.490292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:60145 (local address 127.9.225.3:42447)
0809 00:52:21.490888 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:21.490903 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:21.497408 (+ 6505us) server_negotiation.cc:407] Connection header received
0809 00:52:21.497920 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:21.497928 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:21.498036 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:21.498131 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:21.499544 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.501268 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.504732 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.507308 (+ 2576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.507572 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:21.507717 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:21.509742 (+ 2025us) server_negotiation.cc:299] Negotiation successful
0809 00:52:21.510184 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260809 00:52:21.510744 13347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70373523fc9f4c87ab2d9098c24b6251" candidate_uuid: "80d1702c61c443aaa1a1b664d99d0f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1a6f3c288b04133af66a94d252e8ced" is_pre_election: true
I20260809 00:52:21.511185 13347 raft_consensus.cc:2454] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80d1702c61c443aaa1a1b664d99d0f93 in term 0.
I20260809 00:52:21.511785 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:21.489504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:42447 (local address 127.9.225.2:60145)
0809 00:52:21.497249 (+ 7745us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:21.497269 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:21.497544 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:21.498402 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:21.498407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:21.498435 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:21.499149 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:21.499167 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.501512 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.501518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:21.504502 (+ 2984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:21.504523 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.509192 (+ 4669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.509202 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:21.509542 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:21.509574 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 00:52:21.509702 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 00:52:21.509874 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7608,"mutex_wait_us":363,"thread_start_us":7313,"threads_started":1}
I20260809 00:52:21.511933 13455 leader_election.cc:302] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80d1702c61c443aaa1a1b664d99d0f93, c1a6f3c288b04133af66a94d252e8ced; no voters:
I20260809 00:52:21.512467 13880 raft_consensus.cc:2781] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:52:21.512661 13880 raft_consensus.cc:482] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:52:21.512885 13880 raft_consensus.cc:3037] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:21.513809 13762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70373523fc9f4c87ab2d9098c24b6251" candidate_uuid: "80d1702c61c443aaa1a1b664d99d0f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7d14cf3d3e4b6a831203886c13359e" is_pre_election: true
I20260809 00:52:21.514897 13762 raft_consensus.cc:2454] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80d1702c61c443aaa1a1b664d99d0f93 in term 0.
I20260809 00:52:21.518718 13880 raft_consensus.cc:504] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:21.521056 13880 leader_election.cc:288] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [CANDIDATE]: Term 1 election: Requested vote from peers c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995), 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3:42447)
I20260809 00:52:21.521970 13347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70373523fc9f4c87ab2d9098c24b6251" candidate_uuid: "80d1702c61c443aaa1a1b664d99d0f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1a6f3c288b04133af66a94d252e8ced"
I20260809 00:52:21.522254 13762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70373523fc9f4c87ab2d9098c24b6251" candidate_uuid: "80d1702c61c443aaa1a1b664d99d0f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7d14cf3d3e4b6a831203886c13359e"
I20260809 00:52:21.522469 13347 raft_consensus.cc:3037] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:21.522737 13762 raft_consensus.cc:3037] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:21.527598 13762 raft_consensus.cc:2454] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80d1702c61c443aaa1a1b664d99d0f93 in term 1.
I20260809 00:52:21.527580 13347 raft_consensus.cc:2454] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80d1702c61c443aaa1a1b664d99d0f93 in term 1.
I20260809 00:52:21.528427 13455 leader_election.cc:302] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4f7d14cf3d3e4b6a831203886c13359e, 80d1702c61c443aaa1a1b664d99d0f93, c1a6f3c288b04133af66a94d252e8ced; no voters:
I20260809 00:52:21.529119 13880 raft_consensus.cc:2781] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:21.530174 13880 raft_consensus.cc:686] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 LEADER]: Becoming Leader. State: Replica: 80d1702c61c443aaa1a1b664d99d0f93, State: Running, Role: LEADER
I20260809 00:52:21.531196 13880 consensus_queue.cc:237] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:21.536723 13078 catalog_manager.cc:5495] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2). New cstate: current_term: 1 leader_uuid: "80d1702c61c443aaa1a1b664d99d0f93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } health_report { overall_health: UNKNOWN } } }
I20260809 00:52:21.605468 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:21.594076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.9.225.2:39775)
0809 00:52:21.595498 (+ 1422us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:21.595502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:21.595542 (+ 40us) server_negotiation.cc:407] Connection header received
0809 00:52:21.595648 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:21.595652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:21.595738 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:21.595970 (+ 232us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:21.596934 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.598395 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.600673 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.601846 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.602111 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:21.603320 (+ 1209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:21.604274 (+ 954us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:21.604536 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:21.604641 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 00:52:21.604753 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1249,"thread_start_us":1010,"threads_started":1}
I20260809 00:52:21.635023 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:21.626053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:42447 (local address 127.9.225.2:35981)
0809 00:52:21.627298 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:21.627316 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:21.627527 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:21.628032 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:21.628036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:21.628046 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 00:52:21.628434 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:21.628444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.630483 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.630488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:21.632023 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:21.632041 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.633596 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.633602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:21.633767 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:21.633844 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 00:52:21.634392 (+ 548us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 00:52:21.634402 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 00:52:21.634467 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 00:52:21.634596 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"thread_start_us":623,"threads_started":1}
I20260809 00:52:21.635123 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:21.625876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:35981 (local address 127.9.225.3:42447)
0809 00:52:21.626223 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:21.626230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:21.627448 (+ 1218us) server_negotiation.cc:407] Connection header received
0809 00:52:21.627764 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:21.627772 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:21.627875 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:21.628011 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:21.628627 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.630259 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.632223 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.633190 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.633402 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:21.634084 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:21.634278 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:21.634383 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:21.634527 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 00:52:21.634689 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260809 00:52:21.647805 13762 raft_consensus.cc:1261] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Refusing update from remote peer 80d1702c61c443aaa1a1b664d99d0f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:21.648738 13347 raft_consensus.cc:1261] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Refusing update from remote peer 80d1702c61c443aaa1a1b664d99d0f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:21.649072 13944 consensus_queue.cc:1035] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:21.649996 13880 consensus_queue.cc:1035] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:21.656888 13905 mvcc.cc:205] Tried to move back new op lower bound from 7316425693779423232 to 7316425693318705152. Current Snapshot: MvccSnapshot[applied={T|T < 7316425693779423232}]
I20260809 00:52:21.729096 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:21.718818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:51405 (local address 127.9.225.2:39775)
0809 00:52:21.719054 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:21.719061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:21.721544 (+ 2483us) server_negotiation.cc:407] Connection header received
0809 00:52:21.721840 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:21.721847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:21.721949 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:21.722010 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:21.723153 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.724500 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.726697 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.727356 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.727516 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:21.728258 (+ 742us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:21.728409 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:21.728483 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:21.728694 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 00:52:21.728786 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 00:52:21.729741 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:21.719936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:39775 (local address 127.9.225.3:51405)
0809 00:52:21.721434 (+ 1498us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:21.721460 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:21.721661 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:21.722353 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:21.722358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:21.722373 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 00:52:21.722935 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:21.722951 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.724751 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.724760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:21.726505 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:21.726522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:21.727668 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:21.727676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:21.727977 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:21.728049 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 00:52:21.728558 (+ 509us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 00:52:21.728566 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 00:52:21.728667 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 00:52:21.728829 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1230,"thread_start_us":2284,"threads_started":1}
W20260809 00:52:21.731796 13905 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 00:52:23.394254 13082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:23.394652 13082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:23.401778 13082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866: Sending DeleteTablet for 3 replicas of tablet 70373523fc9f4c87ab2d9098c24b6251
I20260809 00:52:23.402968 13527 tablet_service.cc:1514] Processing DeleteTablet for tablet 70373523fc9f4c87ab2d9098c24b6251 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:23 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:56972
I20260809 00:52:23.403362 13732 tablet_service.cc:1514] Processing DeleteTablet for tablet 70373523fc9f4c87ab2d9098c24b6251 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:23 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:33392
I20260809 00:52:23.403488 13311 tablet_service.cc:1514] Processing DeleteTablet for tablet 70373523fc9f4c87ab2d9098c24b6251 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:23 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:42840
I20260809 00:52:23.403720 14012 tablet_replica.cc:330] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93: stopping tablet replica
I20260809 00:52:23.403945 14012 raft_consensus.cc:2229] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:52:23.404624 14012 raft_consensus.cc:2258] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:23.406628 14012 ts_tablet_manager.cc:1905] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:23.416684 14012 ts_tablet_manager.cc:1918] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 00:52:23.417093 14012 log.cc:1196] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/70373523fc9f4c87ab2d9098c24b6251
I20260809 00:52:23.417676 14015 tablet_replica.cc:330] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced: stopping tablet replica
I20260809 00:52:23.418123 14015 raft_consensus.cc:2229] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:23.418586 14015 raft_consensus.cc:2258] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:23.419376 14014 tablet_replica.cc:330] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e: stopping tablet replica
I20260809 00:52:23.419638 14014 raft_consensus.cc:2229] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:23.419909 14014 raft_consensus.cc:2258] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:23.420913 14015 ts_tablet_manager.cc:1905] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:23.422135 14012 ts_tablet_manager.cc:1939] T 70373523fc9f4c87ab2d9098c24b6251 P 80d1702c61c443aaa1a1b664d99d0f93: Deleting consensus metadata
I20260809 00:52:23.425756 13046 catalog_manager.cc:4841] TS 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2:39775): tablet 70373523fc9f4c87ab2d9098c24b6251 (table test-table [id=6af3df203cd946e1a51d7ebaa1d00a74]) successfully deleted
I20260809 00:52:23.428359 14015 ts_tablet_manager.cc:1918] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 00:52:23.428537 14015 log.cc:1196] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/70373523fc9f4c87ab2d9098c24b6251
I20260809 00:52:23.429155 14015 ts_tablet_manager.cc:1939] T 70373523fc9f4c87ab2d9098c24b6251 P c1a6f3c288b04133af66a94d252e8ced: Deleting consensus metadata
I20260809 00:52:23.430171 14014 ts_tablet_manager.cc:1905] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:23.437112 14014 ts_tablet_manager.cc:1918] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 00:52:23.437527 14014 log.cc:1196] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/70373523fc9f4c87ab2d9098c24b6251
I20260809 00:52:23.438215 14014 ts_tablet_manager.cc:1939] T 70373523fc9f4c87ab2d9098c24b6251 P 4f7d14cf3d3e4b6a831203886c13359e: Deleting consensus metadata
I20260809 00:52:23.439635 13048 catalog_manager.cc:4841] TS 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3:42447): tablet 70373523fc9f4c87ab2d9098c24b6251 (table test-table [id=6af3df203cd946e1a51d7ebaa1d00a74]) successfully deleted
I20260809 00:52:23.441768 13049 catalog_manager.cc:4841] TS c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995): tablet 70373523fc9f4c87ab2d9098c24b6251 (table test-table [id=6af3df203cd946e1a51d7ebaa1d00a74]) successfully deleted
Aug 09 00:52:23 dist-test-slave-j2vl krb5kdc[12998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 00:52:23 dist-test-slave-j2vl krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:23.496428 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:23.480098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.9.225.62:44933)
0809 00:52:23.480948 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:23.480953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:23.480980 (+ 27us) server_negotiation.cc:407] Connection header received
0809 00:52:23.481174 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:23.481178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:23.481247 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:23.481432 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:23.483064 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.484039 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.485459 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.485902 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.486030 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:23.490400 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:23.490407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:23.490410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:23.490478 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:23.493584 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:23.494474 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:23.494479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:23.494481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:23.494647 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:23.495027 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:23.495030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:23.495032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:23.495325 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:23.495522 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:23.495807 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 00:52:23.496018 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":570,"threads_started":1}
I20260809 00:52:23.501076 13082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:52:23.501891 13082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:52:23.510108 13311 tablet_service.cc:1467] Processing CreateTablet for tablet d10016384829470187558f30cb015d3b (DEFAULT_TABLE table=test-table [id=53329210346744b991da6939aa7a81b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:23.510130 13527 tablet_service.cc:1467] Processing CreateTablet for tablet d10016384829470187558f30cb015d3b (DEFAULT_TABLE table=test-table [id=53329210346744b991da6939aa7a81b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:23.510491 13732 tablet_service.cc:1467] Processing CreateTablet for tablet d10016384829470187558f30cb015d3b (DEFAULT_TABLE table=test-table [id=53329210346744b991da6939aa7a81b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:23.510643 13527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10016384829470187558f30cb015d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:23.510668 13311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10016384829470187558f30cb015d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:23.510942 13732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10016384829470187558f30cb015d3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:23.516579 14031 tablet_bootstrap.cc:492] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e: Bootstrap starting.
I20260809 00:52:23.519136 14032 tablet_bootstrap.cc:492] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93: Bootstrap starting.
I20260809 00:52:23.519595 14031 tablet_bootstrap.cc:654] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:23.522837 14037 tablet_bootstrap.cc:492] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced: Bootstrap starting.
I20260809 00:52:23.523416 14032 tablet_bootstrap.cc:654] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:23.524907 14031 tablet_bootstrap.cc:492] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e: No bootstrap required, opened a new log
I20260809 00:52:23.525255 14031 ts_tablet_manager.cc:1397] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260809 00:52:23.525942 14037 tablet_bootstrap.cc:654] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:23.527191 14031 raft_consensus.cc:348] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.527906 14031 raft_consensus.cc:374] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:23.528178 14031 raft_consensus.cc:729] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d14cf3d3e4b6a831203886c13359e, State: Initialized, Role: FOLLOWER
I20260809 00:52:23.528731 14031 consensus_queue.cc:260] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.529628 14031 ts_tablet_manager.cc:1428] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260809 00:52:23.530176 14037 tablet_bootstrap.cc:492] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced: No bootstrap required, opened a new log
I20260809 00:52:23.530459 14037 ts_tablet_manager.cc:1397] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 00:52:23.531746 14032 tablet_bootstrap.cc:492] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93: No bootstrap required, opened a new log
I20260809 00:52:23.531627 14037 raft_consensus.cc:348] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.531945 14037 raft_consensus.cc:374] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:23.532152 14032 ts_tablet_manager.cc:1397] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260809 00:52:23.532186 14037 raft_consensus.cc:729] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1a6f3c288b04133af66a94d252e8ced, State: Initialized, Role: FOLLOWER
I20260809 00:52:23.532594 14037 consensus_queue.cc:260] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.533385 14032 raft_consensus.cc:348] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.533699 14032 raft_consensus.cc:374] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:23.533970 14032 raft_consensus.cc:729] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d1702c61c443aaa1a1b664d99d0f93, State: Initialized, Role: FOLLOWER
I20260809 00:52:23.534412 14032 consensus_queue.cc:260] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.534811 14037 ts_tablet_manager.cc:1428] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260809 00:52:23.535187 14032 ts_tablet_manager.cc:1428] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 00:52:23.591358 14043 raft_consensus.cc:482] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:52:23.591813 14043 raft_consensus.cc:504] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.593777 14043 leader_election.cc:288] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2:39775), 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3:42447)
I20260809 00:52:23.614957 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:23.594173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:48843 (local address 127.9.225.2:39775)
0809 00:52:23.606918 (+ 12745us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:23.606926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:23.606953 (+ 27us) server_negotiation.cc:407] Connection header received
0809 00:52:23.607039 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:23.607043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:23.607106 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:23.607233 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:23.607994 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.609094 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.612010 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.613599 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.613811 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:23.613976 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:23.614350 (+ 374us) server_negotiation.cc:299] Negotiation successful
0809 00:52:23.614523 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12559,"thread_start_us":12395,"threads_started":1}
I20260809 00:52:23.615038 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:23.594158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:39775 (local address 127.9.225.1:48843)
0809 00:52:23.606384 (+ 12226us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:23.606413 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:23.606619 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:23.607262 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:23.607267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:23.607307 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:23.607749 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:23.607763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.609254 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.609257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:23.611762 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:23.611779 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.613977 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.613983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:23.614157 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:23.614186 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 00:52:23.614282 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 00:52:23.614456 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12053,"thread_start_us":13278,"threads_started":1}
I20260809 00:52:23.615612 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:23.594209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:42447 (local address 127.9.225.1:38747)
0809 00:52:23.608088 (+ 13879us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:23.608118 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:23.608254 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:23.608629 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:23.608633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:23.608657 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:23.609043 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:23.609052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.610371 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.610376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:23.612499 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:23.612511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.614602 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.614608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:23.614920 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:23.614954 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 00:52:23.615070 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 00:52:23.615218 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13690,"thread_start_us":11873,"threads_started":1}
I20260809 00:52:23.615655 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:23.594441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:38747 (local address 127.9.225.3:42447)
0809 00:52:23.606047 (+ 11606us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:23.606057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:23.608225 (+ 2168us) server_negotiation.cc:407] Connection header received
0809 00:52:23.608366 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:23.608374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:23.608462 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:23.608575 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:23.609229 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.610205 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.612705 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.614331 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.614477 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:23.614592 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:23.615137 (+ 545us) server_negotiation.cc:299] Negotiation successful
0809 00:52:23.615313 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11423,"thread_start_us":11235,"threads_started":1}
I20260809 00:52:23.616145 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10016384829470187558f30cb015d3b" candidate_uuid: "c1a6f3c288b04133af66a94d252e8ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d1702c61c443aaa1a1b664d99d0f93" is_pre_election: true
I20260809 00:52:23.616460 13552 raft_consensus.cc:2454] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1a6f3c288b04133af66a94d252e8ced in term 0.
I20260809 00:52:23.616443 13762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10016384829470187558f30cb015d3b" candidate_uuid: "c1a6f3c288b04133af66a94d252e8ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7d14cf3d3e4b6a831203886c13359e" is_pre_election: true
I20260809 00:52:23.616715 13762 raft_consensus.cc:2454] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1a6f3c288b04133af66a94d252e8ced in term 0.
I20260809 00:52:23.617220 13225 leader_election.cc:302] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80d1702c61c443aaa1a1b664d99d0f93, c1a6f3c288b04133af66a94d252e8ced; no voters:
I20260809 00:52:23.617880 14043 raft_consensus.cc:2781] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:52:23.618047 14043 raft_consensus.cc:482] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:52:23.618193 14043 raft_consensus.cc:3037] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:23.620800 14043 raft_consensus.cc:504] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.621847 14043 leader_election.cc:288] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [CANDIDATE]: Term 1 election: Requested vote from peers 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2:39775), 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3:42447)
I20260809 00:52:23.622522 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10016384829470187558f30cb015d3b" candidate_uuid: "c1a6f3c288b04133af66a94d252e8ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d1702c61c443aaa1a1b664d99d0f93"
I20260809 00:52:23.622545 13762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10016384829470187558f30cb015d3b" candidate_uuid: "c1a6f3c288b04133af66a94d252e8ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7d14cf3d3e4b6a831203886c13359e"
I20260809 00:52:23.622782 13762 raft_consensus.cc:3037] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:23.622772 13552 raft_consensus.cc:3037] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:23.625236 13552 raft_consensus.cc:2454] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1a6f3c288b04133af66a94d252e8ced in term 1.
I20260809 00:52:23.625609 13762 raft_consensus.cc:2454] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1a6f3c288b04133af66a94d252e8ced in term 1.
I20260809 00:52:23.625949 13225 leader_election.cc:302] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80d1702c61c443aaa1a1b664d99d0f93, c1a6f3c288b04133af66a94d252e8ced; no voters:
I20260809 00:52:23.626368 14043 raft_consensus.cc:2781] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:23.627064 14043 raft_consensus.cc:686] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 1 LEADER]: Becoming Leader. State: Replica: c1a6f3c288b04133af66a94d252e8ced, State: Running, Role: LEADER
I20260809 00:52:23.627585 14043 consensus_queue.cc:237] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.632371 13078 catalog_manager.cc:5495] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced reported cstate change: term changed from 0 to 1, leader changed from <none> to c1a6f3c288b04133af66a94d252e8ced (127.9.225.1). New cstate: current_term: 1 leader_uuid: "c1a6f3c288b04133af66a94d252e8ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } health_report { overall_health: UNKNOWN } } }
I20260809 00:52:23.670815 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:23.662658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42854 (local address 127.9.225.1:32995)
0809 00:52:23.663547 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:23.663552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:23.663578 (+ 26us) server_negotiation.cc:407] Connection header received
0809 00:52:23.663681 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:23.663686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:23.663769 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:23.663976 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:23.664558 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.665661 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.667428 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.668242 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.668368 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:23.669137 (+ 769us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:23.669988 (+ 851us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:23.670146 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:23.670262 (+ 116us) server_negotiation.cc:299] Negotiation successful
0809 00:52:23.670345 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":561,"threads_started":1}
I20260809 00:52:23.695585 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:23.686926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:47235 (local address 127.9.225.3:42447)
0809 00:52:23.687134 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:23.687141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:23.688444 (+ 1303us) server_negotiation.cc:407] Connection header received
0809 00:52:23.688645 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:23.688651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:23.688751 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:23.688864 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:23.689919 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.691147 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.693172 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.693922 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.694088 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:23.694699 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:23.694886 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:23.694985 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:23.695146 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 00:52:23.695242 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 00:52:23.695749 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:23.686583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:42447 (local address 127.9.225.1:47235)
0809 00:52:23.688247 (+ 1664us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:23.688267 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:23.688504 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:23.688889 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:23.688892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:23.688902 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 00:52:23.689619 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:23.689633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.691351 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.691355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:23.692989 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:23.693006 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.694189 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.694195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:23.694409 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:23.694491 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 00:52:23.694995 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 00:52:23.695004 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 00:52:23.695070 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 00:52:23.695202 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1414,"thread_start_us":1001,"threads_started":1}
I20260809 00:52:23.709206 13552 raft_consensus.cc:1261] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Refusing update from remote peer c1a6f3c288b04133af66a94d252e8ced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:23.709506 13762 raft_consensus.cc:1261] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Refusing update from remote peer c1a6f3c288b04133af66a94d252e8ced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:23.710502 14043 consensus_queue.cc:1035] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:23.711087 14048 consensus_queue.cc:1035] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:23.752182 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:23.742992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:56781 (local address 127.9.225.1:32995)
0809 00:52:23.743180 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:23.743185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:23.744483 (+ 1298us) server_negotiation.cc:407] Connection header received
0809 00:52:23.744728 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:23.744734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:23.744822 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:23.744927 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:23.745929 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.747235 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.749545 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.750249 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.750400 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:23.751253 (+ 853us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:23.751386 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:23.751498 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:23.751683 (+ 185us) server_negotiation.cc:299] Negotiation successful
0809 00:52:23.751795 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 00:52:23.752236 14065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:23.742624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:32995 (local address 127.9.225.3:56781)
0809 00:52:23.744171 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:23.744193 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:23.744570 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:23.744962 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:23.744966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:23.744980 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 00:52:23.745583 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:23.745599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.747386 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.747390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:23.749308 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:23.749352 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.750648 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.750660 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:23.750952 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:23.751021 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 00:52:23.751502 (+ 481us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 00:52:23.751518 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 00:52:23.751614 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 00:52:23.751801 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1212,"thread_start_us":845,"threads_started":1}
I20260809 00:52:23.802330 13081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:23.802858 13081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:23.809063 13081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866: Sending DeleteTablet for 3 replicas of tablet d10016384829470187558f30cb015d3b
I20260809 00:52:23.811117 13311 tablet_service.cc:1514] Processing DeleteTablet for tablet d10016384829470187558f30cb015d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:23 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:42840
I20260809 00:52:23.810933 13732 tablet_service.cc:1514] Processing DeleteTablet for tablet d10016384829470187558f30cb015d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:23 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:33392
I20260809 00:52:23.811812 13527 tablet_service.cc:1514] Processing DeleteTablet for tablet d10016384829470187558f30cb015d3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:23 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:56972
I20260809 00:52:23.811990 14015 tablet_replica.cc:330] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced: stopping tablet replica
I20260809 00:52:23.812037 14014 tablet_replica.cc:330] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e: stopping tablet replica
I20260809 00:52:23.812340 14012 tablet_replica.cc:330] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93: stopping tablet replica
I20260809 00:52:23.812366 14015 raft_consensus.cc:2229] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:52:23.812652 14012 raft_consensus.cc:2229] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:23.812597 14014 raft_consensus.cc:2229] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:23.813107 14012 raft_consensus.cc:2258] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:23.813421 14014 raft_consensus.cc:2258] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:23.813541 14015 raft_consensus.cc:2258] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:23.815331 14012 ts_tablet_manager.cc:1905] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:23.815384 14015 ts_tablet_manager.cc:1905] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:23.815656 13081 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 00:52:23.816043 14014 ts_tablet_manager.cc:1905] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 00:52:23.817039 13081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:52:23.827188 14014 ts_tablet_manager.cc:1918] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:52:23.827205 14015 ts_tablet_manager.cc:1918] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:52:23.827699 14015 log.cc:1196] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/d10016384829470187558f30cb015d3b
I20260809 00:52:23.827986 14014 log.cc:1196] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/d10016384829470187558f30cb015d3b
I20260809 00:52:23.828646 14015 ts_tablet_manager.cc:1939] T d10016384829470187558f30cb015d3b P c1a6f3c288b04133af66a94d252e8ced: Deleting consensus metadata
I20260809 00:52:23.829077 14014 ts_tablet_manager.cc:1939] T d10016384829470187558f30cb015d3b P 4f7d14cf3d3e4b6a831203886c13359e: Deleting consensus metadata
I20260809 00:52:23.831027 13048 catalog_manager.cc:4841] TS 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3:42447): tablet d10016384829470187558f30cb015d3b (table test-table [id=53329210346744b991da6939aa7a81b4]) successfully deleted
I20260809 00:52:23.831066 13049 catalog_manager.cc:4841] TS c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995): tablet d10016384829470187558f30cb015d3b (table test-table [id=53329210346744b991da6939aa7a81b4]) successfully deleted
I20260809 00:52:23.836076 14012 ts_tablet_manager.cc:1918] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:52:23.836692 14012 log.cc:1196] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/d10016384829470187558f30cb015d3b
I20260809 00:52:23.836884 13527 tablet_service.cc:1467] Processing CreateTablet for tablet 07c57c1d8f994db591e511eb499b7dbc (DEFAULT_TABLE table=test-table [id=0101016dfb234b5ab0d1627b85b9e7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:23.836969 13311 tablet_service.cc:1467] Processing CreateTablet for tablet 07c57c1d8f994db591e511eb499b7dbc (DEFAULT_TABLE table=test-table [id=0101016dfb234b5ab0d1627b85b9e7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:23.837797 14012 ts_tablet_manager.cc:1939] T d10016384829470187558f30cb015d3b P 80d1702c61c443aaa1a1b664d99d0f93: Deleting consensus metadata
I20260809 00:52:23.837824 13311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c57c1d8f994db591e511eb499b7dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:23.837957 13527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c57c1d8f994db591e511eb499b7dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:23.840163 13046 catalog_manager.cc:4841] TS 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2:39775): tablet d10016384829470187558f30cb015d3b (table test-table [id=53329210346744b991da6939aa7a81b4]) successfully deleted
I20260809 00:52:23.844092 13732 tablet_service.cc:1467] Processing CreateTablet for tablet 07c57c1d8f994db591e511eb499b7dbc (DEFAULT_TABLE table=test-table [id=0101016dfb234b5ab0d1627b85b9e7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:23.845257 13732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c57c1d8f994db591e511eb499b7dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:23.850661 14032 tablet_bootstrap.cc:492] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93: Bootstrap starting.
I20260809 00:52:23.851199 14037 tablet_bootstrap.cc:492] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced: Bootstrap starting.
I20260809 00:52:23.854602 14032 tablet_bootstrap.cc:654] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:23.856667 14037 tablet_bootstrap.cc:654] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:23.865065 14032 tablet_bootstrap.cc:492] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93: No bootstrap required, opened a new log
I20260809 00:52:23.865707 14032 ts_tablet_manager.cc:1397] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20260809 00:52:23.867120 14031 tablet_bootstrap.cc:492] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e: Bootstrap starting.
I20260809 00:52:23.867440 14032 raft_consensus.cc:348] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.867905 14032 raft_consensus.cc:374] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:23.868090 14032 raft_consensus.cc:729] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d1702c61c443aaa1a1b664d99d0f93, State: Initialized, Role: FOLLOWER
I20260809 00:52:23.868778 14032 consensus_queue.cc:260] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.870307 14032 ts_tablet_manager.cc:1428] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260809 00:52:23.871232 14031 tablet_bootstrap.cc:654] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:23.872507 14037 tablet_bootstrap.cc:492] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced: No bootstrap required, opened a new log
I20260809 00:52:23.873538 14037 ts_tablet_manager.cc:1397] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.017s
I20260809 00:52:23.875349 14037 raft_consensus.cc:348] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.876803 14037 raft_consensus.cc:374] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:23.877007 14037 raft_consensus.cc:729] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1a6f3c288b04133af66a94d252e8ced, State: Initialized, Role: FOLLOWER
I20260809 00:52:23.877490 14037 consensus_queue.cc:260] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.878338 14037 ts_tablet_manager.cc:1428] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260809 00:52:23.879426 14031 tablet_bootstrap.cc:492] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e: No bootstrap required, opened a new log
I20260809 00:52:23.879681 14031 ts_tablet_manager.cc:1397] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.008s
I20260809 00:52:23.880853 14031 raft_consensus.cc:348] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.881309 14031 raft_consensus.cc:374] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:23.881575 14031 raft_consensus.cc:729] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7d14cf3d3e4b6a831203886c13359e, State: Initialized, Role: FOLLOWER
I20260809 00:52:23.882050 14031 consensus_queue.cc:260] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.884078 14031 ts_tablet_manager.cc:1428] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260809 00:52:23.907423 14003 raft_consensus.cc:482] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:52:23.907821 14003 raft_consensus.cc:504] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.909034 14003 leader_election.cc:288] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995), 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3:42447)
I20260809 00:52:23.909672 13345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57c1d8f994db591e511eb499b7dbc" candidate_uuid: "80d1702c61c443aaa1a1b664d99d0f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1a6f3c288b04133af66a94d252e8ced" is_pre_election: true
I20260809 00:52:23.910095 13345 raft_consensus.cc:2454] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80d1702c61c443aaa1a1b664d99d0f93 in term 0.
I20260809 00:52:23.909924 13762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57c1d8f994db591e511eb499b7dbc" candidate_uuid: "80d1702c61c443aaa1a1b664d99d0f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7d14cf3d3e4b6a831203886c13359e" is_pre_election: true
I20260809 00:52:23.910388 13762 raft_consensus.cc:2454] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80d1702c61c443aaa1a1b664d99d0f93 in term 0.
I20260809 00:52:23.910764 13455 leader_election.cc:302] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80d1702c61c443aaa1a1b664d99d0f93, c1a6f3c288b04133af66a94d252e8ced; no voters:
I20260809 00:52:23.911134 14003 raft_consensus.cc:2781] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:52:23.911317 14003 raft_consensus.cc:482] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:52:23.911459 14003 raft_consensus.cc:3037] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:23.914803 14003 raft_consensus.cc:504] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.915745 14003 leader_election.cc:288] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [CANDIDATE]: Term 1 election: Requested vote from peers c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995), 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3:42447)
I20260809 00:52:23.916507 13345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57c1d8f994db591e511eb499b7dbc" candidate_uuid: "80d1702c61c443aaa1a1b664d99d0f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1a6f3c288b04133af66a94d252e8ced"
I20260809 00:52:23.916651 13762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57c1d8f994db591e511eb499b7dbc" candidate_uuid: "80d1702c61c443aaa1a1b664d99d0f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7d14cf3d3e4b6a831203886c13359e"
I20260809 00:52:23.917048 13345 raft_consensus.cc:3037] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:23.917057 13762 raft_consensus.cc:3037] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:23.920598 13762 raft_consensus.cc:2454] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80d1702c61c443aaa1a1b664d99d0f93 in term 1.
I20260809 00:52:23.921411 13454 leader_election.cc:302] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7d14cf3d3e4b6a831203886c13359e, 80d1702c61c443aaa1a1b664d99d0f93; no voters:
I20260809 00:52:23.921903 14003 raft_consensus.cc:2781] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:23.922747 13345 raft_consensus.cc:2454] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80d1702c61c443aaa1a1b664d99d0f93 in term 1.
I20260809 00:52:23.922950 14003 raft_consensus.cc:686] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 LEADER]: Becoming Leader. State: Replica: 80d1702c61c443aaa1a1b664d99d0f93, State: Running, Role: LEADER
I20260809 00:52:23.923504 14003 consensus_queue.cc:237] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } }
I20260809 00:52:23.927753 13082 catalog_manager.cc:5495] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2). New cstate: current_term: 1 leader_uuid: "80d1702c61c443aaa1a1b664d99d0f93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80d1702c61c443aaa1a1b664d99d0f93" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 } health_report { overall_health: UNKNOWN } } }
I20260809 00:52:23.969898 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:23.962058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.9.225.2:39775)
0809 00:52:23.962298 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:23.962303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:23.962983 (+ 680us) server_negotiation.cc:407] Connection header received
0809 00:52:23.963274 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:23.963278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:23.963433 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:23.963540 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:23.964014 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.965226 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.967407 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:23.968104 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:23.968259 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:23.968928 (+ 669us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:23.969085 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:23.969196 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:23.969390 (+ 194us) server_negotiation.cc:299] Negotiation successful
0809 00:52:23.969485 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 00:52:23.985508 13762 raft_consensus.cc:1261] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Refusing update from remote peer 80d1702c61c443aaa1a1b664d99d0f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:23.986246 13345 raft_consensus.cc:1261] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Refusing update from remote peer 80d1702c61c443aaa1a1b664d99d0f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:23.986405 14072 consensus_queue.cc:1035] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7d14cf3d3e4b6a831203886c13359e" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 42447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:23.987386 14003 consensus_queue.cc:1035] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1a6f3c288b04133af66a94d252e8ced" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 32995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 00:52:24.017796 13974 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 00:52:25.644712 13082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:25.645146 13082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:25.651930 13082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa5b03483bdd438f92f0915000f86866: Sending DeleteTablet for 3 replicas of tablet 07c57c1d8f994db591e511eb499b7dbc
I20260809 00:52:25.653280 13311 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c57c1d8f994db591e511eb499b7dbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:25 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:42840
I20260809 00:52:25.653254 13527 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c57c1d8f994db591e511eb499b7dbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:25 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:56972
I20260809 00:52:25.654587 13732 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c57c1d8f994db591e511eb499b7dbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:25 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:33392
I20260809 00:52:25.655154 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13201
I20260809 00:52:25.657501 14128 tablet_replica.cc:330] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93: stopping tablet replica
I20260809 00:52:25.657897 14128 raft_consensus.cc:2229] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:52:25.658613 14128 raft_consensus.cc:2258] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:25.660305 14128 ts_tablet_manager.cc:1905] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:25.671087 14131 tablet_replica.cc:330] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e: stopping tablet replica
I20260809 00:52:25.671630 14131 raft_consensus.cc:2229] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:25.672225 14127 tablet_replica.cc:330] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced: stopping tablet replica
I20260809 00:52:25.672211 14131 raft_consensus.cc:2258] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:25.673051 14127 raft_consensus.cc:2229] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:25.673324 14127 raft_consensus.cc:2258] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:25.673626 14131 ts_tablet_manager.cc:1905] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:25.674379 14127 ts_tablet_manager.cc:1905] T 07c57c1d8f994db591e511eb499b7dbc P c1a6f3c288b04133af66a94d252e8ced: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 00:52:25 dist-test-slave-j2vl krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:25.693930 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:25.658041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.9.225.1:32995)
0809 00:52:25.659567 (+ 1526us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:25.659577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:25.660621 (+ 1044us) server_negotiation.cc:407] Connection header received
0809 00:52:25.660719 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:25.660729 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:25.660922 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:25.661073 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:25.664960 (+ 3887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:25.667383 (+ 2423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:25.670648 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:25.672082 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:25.672264 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:25.684106 (+ 11842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:25.684135 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:25.684170 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:25.684256 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:25.689324 (+ 5068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:25.690650 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:25.690662 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:25.690666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:25.691013 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:25.691538 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:25.691543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:25.691547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:25.692154 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:25.692691 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:25.692957 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 00:52:25.693293 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":1159,"threads_started":1}
W20260809 00:52:25.694867 13408 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49812
W20260809 00:52:25.695958 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13201: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:52:25.696369 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 13201
I20260809 00:52:25.759183 14128 ts_tablet_manager.cc:1918] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 00:52:25.759481 14128 log.cc:1196] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/07c57c1d8f994db591e511eb499b7dbc
I20260809 00:52:25.760212 14128 ts_tablet_manager.cc:1939] T 07c57c1d8f994db591e511eb499b7dbc P 80d1702c61c443aaa1a1b664d99d0f93: Deleting consensus metadata
I20260809 00:52:25.763314 13046 catalog_manager.cc:4841] TS 80d1702c61c443aaa1a1b664d99d0f93 (127.9.225.2:39775): tablet 07c57c1d8f994db591e511eb499b7dbc (table test-table [id=0101016dfb234b5ab0d1627b85b9e7a0]) successfully deleted
I20260809 00:52:25.764715 14131 ts_tablet_manager.cc:1918] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 00:52:25.764993 14131 log.cc:1196] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/07c57c1d8f994db591e511eb499b7dbc
I20260809 00:52:25.765635 14131 ts_tablet_manager.cc:1939] T 07c57c1d8f994db591e511eb499b7dbc P 4f7d14cf3d3e4b6a831203886c13359e: Deleting consensus metadata
I20260809 00:52:25.790973 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13421
W20260809 00:52:25.791139 13049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.225.1:32995 (error 108)
I20260809 00:52:25.794699 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:25.793291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:32995 (local address 127.0.0.1:49826)
0809 00:52:25.794009 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:25.794244 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":363,"threads_started":1}
W20260809 00:52:25.795043 13049 catalog_manager.cc:4584] TS c1a6f3c288b04133af66a94d252e8ced (127.9.225.1:32995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07c57c1d8f994db591e511eb499b7dbc: Network error: Client connection negotiation failed: client connection to 127.9.225.1:32995: connect: Connection refused (error 111)
Aug 09 00:52:25 dist-test-slave-j2vl krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:52:25.811517 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:25.791964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39624 (local address 127.9.225.2:39775)
0809 00:52:25.792904 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:25.792909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:25.792926 (+ 17us) server_negotiation.cc:407] Connection header received
0809 00:52:25.793012 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:25.793021 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:25.793115 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:25.793257 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:25.796093 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:25.796907 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:25.798591 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:25.799232 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:25.799419 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:25.804536 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:25.804562 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:25.804582 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:25.804642 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:25.808310 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:25.809210 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:25.809218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:25.809221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:25.809451 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:25.809852 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:25.809856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:25.809858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:25.810283 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:25.810571 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:25.810878 (+ 307us) server_negotiation.cc:299] Negotiation successful
0809 00:52:25.811140 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":638,"threads_started":1}
W20260809 00:52:25.812252 13617 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39624
W20260809 00:52:25.813103 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13421: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:52:25.813311 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 13421
I20260809 00:52:25.849120 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:25.848419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:32995 (local address 127.0.0.1:49840)
0809 00:52:25.848752 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:25.848886 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260809 00:52:25.850867 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:25.850262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:32995 (local address 127.0.0.1:49852)
0809 00:52:25.850516 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:25.850648 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260809 00:52:25.853016 13048 catalog_manager.cc:4841] TS 4f7d14cf3d3e4b6a831203886c13359e (127.9.225.3:42447): tablet 07c57c1d8f994db591e511eb499b7dbc (table test-table [id=0101016dfb234b5ab0d1627b85b9e7a0]) successfully deleted
I20260809 00:52:25.857318 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13628
Aug 09 00:52:25 dist-test-slave-j2vl krb5kdc[12998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:52:25.878849 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:25.858507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.9.225.3:42447)
0809 00:52:25.859121 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:25.859126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:25.859144 (+ 18us) server_negotiation.cc:407] Connection header received
0809 00:52:25.859203 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:25.859207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:25.859285 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:25.859391 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:25.861136 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:25.862118 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:25.864296 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:25.864766 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:25.864886 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:25.871075 (+ 6189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:25.871120 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:25.871141 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:25.871203 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:25.875075 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:25.876086 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:25.876106 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:25.876109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:25.876326 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:25.876861 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:25.876865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:25.876867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:25.877488 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:25.877800 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:25.878180 (+ 380us) server_negotiation.cc:299] Negotiation successful
0809 00:52:25.878410 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":345,"threads_started":1}
W20260809 00:52:25.879616 13835 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38408
W20260809 00:52:25.880399 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13628: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:52:25.880587 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 13628
I20260809 00:52:25.911707 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:25.911197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:32995 (local address 127.0.0.1:49862)
0809 00:52:25.911431 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:25.911541 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 00:52:25.912868 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:25.912471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:32995 (local address 127.0.0.1:49864)
0809 00:52:25.912645 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:25.912719 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 00:52:25.923851 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13019
I20260809 00:52:25.925272 13147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:26.062829 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:26.062191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:32995 (local address 127.0.0.1:49878)
0809 00:52:26.062450 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:26.062609 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 00:52:26.063309 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 13019
2026-08-09T00:52:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9908 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:52:26 dist-test-slave-j2vl krb5kdc[14144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:52:26 dist-test-slave-j2vl krb5kdc[14144](info): set up 2 sockets
Aug 09 00:52:26 dist-test-slave-j2vl krb5kdc[14144](info): commencing operation
krb5kdc: starting...
W20260809 00:52:27.512213 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.389s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:52:27 dist-test-slave-j2vl krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:52:27.770543 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:37475
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:37475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:28.315609 14163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:28.315969 14163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:28.316138 14163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:28.324465 14163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:28.324630 14163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:28.324755 14163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:28.324857 14163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:28.324901 14163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:52:28.336143 14163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:37475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:37475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:28.337901 14163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:28.339500 14163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:28.354678 14177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:28.354676 14174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:28.354900 14173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:28.355898 14163 server_base.cc:1034] running on GCE node
I20260809 00:52:28.357235 14163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:28.359306 14163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:28.360590 14163 hybrid_clock.cc:648] HybridClock initialized: now 1786236748360540 us; error 47 us; skew 500 ppm
I20260809 00:52:28.361187 14163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:28 dist-test-slave-j2vl krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236748, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:28.365552 14163 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:52:28.367652 14163 webserver.cc:460] Webserver started at http://127.9.225.62:37811/ using document root <none> and password file <none>
I20260809 00:52:28.368582 14163 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:28.368937 14163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:28.369581 14163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:28.373281 14163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "6d5ec2fbfd114afaa7296c231ac7465d"
format_stamp: "Formatted at 2026-08-09 00:52:28 on dist-test-slave-j2vl"
server_key: "3a98a2a3adf74f45c65931f3322bae18"
server_key_iv: "64dcf116f6ca5b8eccf824f6ade1973e"
server_key_version: "encryptionkey@0"
I20260809 00:52:28.374383 14163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "6d5ec2fbfd114afaa7296c231ac7465d"
format_stamp: "Formatted at 2026-08-09 00:52:28 on dist-test-slave-j2vl"
server_key: "3a98a2a3adf74f45c65931f3322bae18"
server_key_iv: "64dcf116f6ca5b8eccf824f6ade1973e"
server_key_version: "encryptionkey@0"
I20260809 00:52:28.381042 14163 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 00:52:28.385917 14190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:28.387698 14163 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:52:28.387972 14163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "6d5ec2fbfd114afaa7296c231ac7465d"
format_stamp: "Formatted at 2026-08-09 00:52:28 on dist-test-slave-j2vl"
server_key: "3a98a2a3adf74f45c65931f3322bae18"
server_key_iv: "64dcf116f6ca5b8eccf824f6ade1973e"
server_key_version: "encryptionkey@0"
I20260809 00:52:28.388234 14163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:28.405164 14163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:28.410513 14163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:28.410878 14163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:28.434073 14163 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:37475
I20260809 00:52:28.434105 14274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:37475 every 8 connection(s)
I20260809 00:52:28.435843 14163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:28.440232 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:28.445941 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 14163
I20260809 00:52:28.446219 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:28.446889 10116 external_mini_cluster.cc:1428] Setting key 10b2888987dd656fec731bd918018432
I20260809 00:52:28.453172 14278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d: Bootstrap starting.
I20260809 00:52:28.456776 14278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:28.458104 14278 log.cc:824] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:28.462864 14278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d: No bootstrap required, opened a new log
Aug 09 00:52:28 dist-test-slave-j2vl krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:28.474814 14278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5ec2fbfd114afaa7296c231ac7465d" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 37475 } }
I20260809 00:52:28.475361 14278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:28.475487 14278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5ec2fbfd114afaa7296c231ac7465d, State: Initialized, Role: FOLLOWER
I20260809 00:52:28.476379 14278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5ec2fbfd114afaa7296c231ac7465d" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 37475 } }
I20260809 00:52:28.476667 14278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:28.476820 14278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:28.477025 14278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:28.478027 14282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:28.449256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.9.225.62:37475)
0809 00:52:28.450483 (+ 1227us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:28.450504 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:28.450564 (+ 60us) server_negotiation.cc:407] Connection header received
0809 00:52:28.452682 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:28.452713 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:28.453238 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:28.453843 (+ 605us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:28.455707 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:28.457389 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:28.459355 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:28.460715 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:28.460948 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:28.467610 (+ 6662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:28.467629 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:28.467666 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:28.467747 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:28.472218 (+ 4471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:28.473487 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:28.473502 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:28.473512 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:28.473781 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:28.474300 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:28.474303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:28.474305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:28.475035 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:28.475321 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:28.475961 (+ 640us) server_negotiation.cc:299] Negotiation successful
0809 00:52:28.476551 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":606,"threads_started":1}
I20260809 00:52:28.480171 14278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5ec2fbfd114afaa7296c231ac7465d" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 37475 } }
I20260809 00:52:28.480947 14278 leader_election.cc:302] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d5ec2fbfd114afaa7296c231ac7465d; no voters:
I20260809 00:52:28.481796 14278 leader_election.cc:288] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:28.481935 14284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:28.482671 14284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [term 1 LEADER]: Becoming Leader. State: Replica: 6d5ec2fbfd114afaa7296c231ac7465d, State: Running, Role: LEADER
I20260809 00:52:28.483320 14284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5ec2fbfd114afaa7296c231ac7465d" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 37475 } }
I20260809 00:52:28.484448 14278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:28.487411 14286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d5ec2fbfd114afaa7296c231ac7465d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5ec2fbfd114afaa7296c231ac7465d" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 37475 } } }
I20260809 00:52:28.487833 14286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:28.488637 14287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d5ec2fbfd114afaa7296c231ac7465d. Latest consensus state: current_term: 1 leader_uuid: "6d5ec2fbfd114afaa7296c231ac7465d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d5ec2fbfd114afaa7296c231ac7465d" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 37475 } } }
I20260809 00:52:28.489032 14287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:28.490706 14294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:28.497722 14294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:28.509689 14294 catalog_manager.cc:1347] Generated new cluster ID: 67bf3896b03547a990d26ccab2ccfd98
I20260809 00:52:28.509851 14294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:28.536547 14294 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:28.538420 14294 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:28.546684 14294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d5ec2fbfd114afaa7296c231ac7465d: Generated new TSK 0
I20260809 00:52:28.547616 14294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:52:28.614372 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:37475
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:29.002816 14314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:29.003159 14314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:29.003331 14314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:29.011137 14314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:29.011327 14314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:29.011534 14314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:52:29.022637 14314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:29.024333 14314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:29.025966 14314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:29.040841 14319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:29.040892 14322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:29.040922 14320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:29.042645 14314 server_base.cc:1034] running on GCE node
I20260809 00:52:29.043414 14314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:29.044559 14314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:29.045828 14314 hybrid_clock.cc:648] HybridClock initialized: now 1786236749045790 us; error 59 us; skew 500 ppm
I20260809 00:52:29.046432 14314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:29 dist-test-slave-j2vl krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236749, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:29.050663 14314 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:52:29.052589 14314 webserver.cc:460] Webserver started at http://127.9.225.1:42089/ using document root <none> and password file <none>
I20260809 00:52:29.053545 14314 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:29.053706 14314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:29.054106 14314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:29.057737 14314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "e38994d1c2994992ad49c9365be6d3aa"
format_stamp: "Formatted at 2026-08-09 00:52:29 on dist-test-slave-j2vl"
server_key: "4d640fe27fc440ff62e97b0b66cc5b45"
server_key_iv: "d4c166c091bc5473e69fbd9b6eb9aa2a"
server_key_version: "encryptionkey@0"
I20260809 00:52:29.058961 14314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "e38994d1c2994992ad49c9365be6d3aa"
format_stamp: "Formatted at 2026-08-09 00:52:29 on dist-test-slave-j2vl"
server_key: "4d640fe27fc440ff62e97b0b66cc5b45"
server_key_iv: "d4c166c091bc5473e69fbd9b6eb9aa2a"
server_key_version: "encryptionkey@0"
I20260809 00:52:29.065928 14314 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260809 00:52:29.070850 14330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:29.072741 14314 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:52:29.072993 14314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "e38994d1c2994992ad49c9365be6d3aa"
format_stamp: "Formatted at 2026-08-09 00:52:29 on dist-test-slave-j2vl"
server_key: "4d640fe27fc440ff62e97b0b66cc5b45"
server_key_iv: "d4c166c091bc5473e69fbd9b6eb9aa2a"
server_key_version: "encryptionkey@0"
I20260809 00:52:29.073271 14314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:29.107759 14314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:29.114992 14314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:29.115500 14314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:29.117108 14314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:29.119328 14314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:29.119473 14314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:29.119673 14314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:29.119752 14314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:29.161393 14314 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:41711
I20260809 00:52:29.161484 14504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:41711 every 8 connection(s)
I20260809 00:52:29.163003 14314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:52:29.164999 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 14314
I20260809 00:52:29.165254 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:52:29.166108 10116 external_mini_cluster.cc:1428] Setting key 674e25c855ee6ad548c351214ce6716f
Aug 09 00:52:29 dist-test-slave-j2vl krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236749, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:29.188021 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:29.166708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:57123 (local address 127.9.225.62:37475)
0809 00:52:29.167263 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:29.167269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:29.168769 (+ 1500us) server_negotiation.cc:407] Connection header received
0809 00:52:29.170875 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:29.170881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:29.170988 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:29.171118 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:29.172996 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:29.174076 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:29.175985 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:29.176422 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:29.176596 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:29.181015 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:29.181021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:29.181024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:29.181067 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:29.183984 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:29.184962 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:29.184970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:29.184973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:29.185123 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:29.185697 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:29.185703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:29.185708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:29.186045 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:29.186420 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:29.187304 (+ 884us) server_negotiation.cc:299] Negotiation successful
0809 00:52:29.187586 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":313,"threads_started":1}
I20260809 00:52:29.189203 14515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:29.167549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:37475 (local address 127.9.225.1:57123)
0809 00:52:29.168573 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:29.168642 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:29.170553 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:29.171413 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:29.171424 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:29.172050 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:29.172697 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:29.172717 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:29.174309 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:29.174313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:29.175786 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:29.175802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:29.176723 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:29.176731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:29.177018 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:29.177967 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:29.178022 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:29.180670 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:29.184227 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:29.184241 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:29.184264 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:29.184806 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:29.185253 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:29.185259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:29.185262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:29.185567 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:29.186377 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:29.186388 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:29.186851 (+ 463us) client_negotiation.cc:769] Sending connection context
0809 00:52:29.187169 (+ 318us) client_negotiation.cc:240] Negotiation successful
0809 00:52:29.187840 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":431,"threads_started":1}
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
I20260809 00:52:29.191478 14510 heartbeater.cc:347] Connected to a master server at 127.9.225.62:37475
I20260809 00:52:29.191912 14510 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:29.192960 14510 heartbeater.cc:511] Master 127.9.225.62:37475 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
I20260809 00:52:29.197139 14220 ts_manager.cc:194] Registered new tserver with Master: e38994d1c2994992ad49c9365be6d3aa (127.9.225.1:41711)
I20260809 00:52:29.199158 14220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:57123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:52:29.226459 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:37475
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:29.600471 14520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:29.600875 14520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:29.601059 14520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:29.608827 14520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:29.609036 14520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:29.609241 14520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:52:29.619899 14520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:29.621610 14520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:29.623103 14520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:29.638549 14528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:29.638546 14527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:29.638655 14530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:29.639583 14520 server_base.cc:1034] running on GCE node
I20260809 00:52:29.640278 14520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:29.641419 14520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:29.642654 14520 hybrid_clock.cc:648] HybridClock initialized: now 1786236749642583 us; error 64 us; skew 500 ppm
I20260809 00:52:29.643046 14520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:29 dist-test-slave-j2vl krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236749, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:29.646626 14520 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:52:29.648451 14520 webserver.cc:460] Webserver started at http://127.9.225.2:34667/ using document root <none> and password file <none>
I20260809 00:52:29.649314 14520 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:29.649547 14520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:29.649923 14520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:29.653301 14520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "f00ccbe8a2814ec2a79f57b6f149b96e"
format_stamp: "Formatted at 2026-08-09 00:52:29 on dist-test-slave-j2vl"
server_key: "4808f88af01877b8525b8b7d34c979f3"
server_key_iv: "88413665316693dfc105cde33682342b"
server_key_version: "encryptionkey@0"
I20260809 00:52:29.654366 14520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "f00ccbe8a2814ec2a79f57b6f149b96e"
format_stamp: "Formatted at 2026-08-09 00:52:29 on dist-test-slave-j2vl"
server_key: "4808f88af01877b8525b8b7d34c979f3"
server_key_iv: "88413665316693dfc105cde33682342b"
server_key_version: "encryptionkey@0"
I20260809 00:52:29.661036 14520 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 00:52:29.666177 14539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:29.668136 14520 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 00:52:29.668399 14520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "f00ccbe8a2814ec2a79f57b6f149b96e"
format_stamp: "Formatted at 2026-08-09 00:52:29 on dist-test-slave-j2vl"
server_key: "4808f88af01877b8525b8b7d34c979f3"
server_key_iv: "88413665316693dfc105cde33682342b"
server_key_version: "encryptionkey@0"
I20260809 00:52:29.668660 14520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:29.684065 14520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:29.689266 14520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:29.689702 14520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:29.691567 14520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:29.693902 14520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:29.694075 14520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:29.694271 14520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:29.694336 14520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:29.735984 14520 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:39571
I20260809 00:52:29.736022 14715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:39571 every 8 connection(s)
I20260809 00:52:29.737548 14520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:52:29.743980 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 14520
I20260809 00:52:29.744225 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:52:29.744707 10116 external_mini_cluster.cc:1428] Setting key 6222d2a0da325d927871a1571ee353d9
Aug 09 00:52:29 dist-test-slave-j2vl krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236749, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:29.764264 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:29.740606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:60369 (local address 127.9.225.62:37475)
0809 00:52:29.741121 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:29.741126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:29.742388 (+ 1262us) server_negotiation.cc:407] Connection header received
0809 00:52:29.744550 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:29.744556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:29.744665 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:29.744810 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:29.747198 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:29.748138 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:29.750431 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:29.750929 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:29.751099 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:29.755905 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:29.755911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:29.755915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:29.755957 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:29.759282 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:29.760758 (+ 1476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:29.760766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:29.760769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:29.761051 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:29.761923 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:29.761930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:29.761932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:29.762392 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:29.762554 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:29.763697 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0809 00:52:29.763976 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":263,"threads_started":1}
I20260809 00:52:29.766188 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:29.741251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:37475 (local address 127.9.225.2:60369)
0809 00:52:29.742171 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:29.742231 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:29.744128 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:29.745110 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:29.745121 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:29.745984 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:29.746750 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:29.746791 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:29.748394 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:29.748400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:29.750159 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:29.750186 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:29.751229 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:29.751238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:29.751514 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:29.752484 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:29.752513 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:29.755557 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:29.759531 (+ 3974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:29.759547 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:29.759575 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:29.760379 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:29.761237 (+ 858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:29.761245 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:29.761252 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:29.761667 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:29.762572 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:29.762584 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:29.763172 (+ 588us) client_negotiation.cc:769] Sending connection context
0809 00:52:29.763567 (+ 395us) client_negotiation.cc:240] Negotiation successful
0809 00:52:29.764315 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":332,"threads_started":1}
I20260809 00:52:29.768471 14716 heartbeater.cc:347] Connected to a master server at 127.9.225.62:37475
I20260809 00:52:29.768918 14716 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:29.769917 14716 heartbeater.cc:511] Master 127.9.225.62:37475 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
I20260809 00:52:29.771862 14220 ts_manager.cc:194] Registered new tserver with Master: f00ccbe8a2814ec2a79f57b6f149b96e (127.9.225.2:39571)
I20260809 00:52:29.772945 14220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:60369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:52:29.805321 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:37475
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:30.184834 14725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:30.185176 14725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:30.185416 14725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:30.193238 14725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:30.193485 14725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:30.193703 14725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:52:30.203402 14510 heartbeater.cc:503] Master 127.9.225.62:37475 was elected leader, sending a full tablet report...
I20260809 00:52:30.204604 14725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:30.206238 14725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:30.207659 14725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:30.222371 14732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:30.222493 14735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:30.222530 14740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:30.224073 14725 server_base.cc:1034] running on GCE node
I20260809 00:52:30.224895 14725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:30.226169 14725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:30.227474 14725 hybrid_clock.cc:648] HybridClock initialized: now 1786236750227358 us; error 87 us; skew 500 ppm
I20260809 00:52:30.227878 14725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:30 dist-test-slave-j2vl krb5kdc[14144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236750, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:30.231719 14725 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:52:30.233561 14725 webserver.cc:460] Webserver started at http://127.9.225.3:34149/ using document root <none> and password file <none>
I20260809 00:52:30.234403 14725 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:30.234576 14725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:30.234956 14725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:30.238379 14725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "2a4954bd74d842e9b74a187504c46a4b"
format_stamp: "Formatted at 2026-08-09 00:52:30 on dist-test-slave-j2vl"
server_key: "981af85c4a645b4ab08f4f61eebbd8d6"
server_key_iv: "5c7da6c7a573c510578adc5e2a4dafec"
server_key_version: "encryptionkey@0"
I20260809 00:52:30.239391 14725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "2a4954bd74d842e9b74a187504c46a4b"
format_stamp: "Formatted at 2026-08-09 00:52:30 on dist-test-slave-j2vl"
server_key: "981af85c4a645b4ab08f4f61eebbd8d6"
server_key_iv: "5c7da6c7a573c510578adc5e2a4dafec"
server_key_version: "encryptionkey@0"
I20260809 00:52:30.245810 14725 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 00:52:30.250195 14750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:30.251995 14725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260809 00:52:30.252264 14725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "2a4954bd74d842e9b74a187504c46a4b"
format_stamp: "Formatted at 2026-08-09 00:52:30 on dist-test-slave-j2vl"
server_key: "981af85c4a645b4ab08f4f61eebbd8d6"
server_key_iv: "5c7da6c7a573c510578adc5e2a4dafec"
server_key_version: "encryptionkey@0"
I20260809 00:52:30.252532 14725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:30.269944 14725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:30.275077 14725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:30.275424 14725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:30.276551 14725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:30.278323 14725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:30.278436 14725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:30.278632 14725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:30.278718 14725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:30.315593 14725 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:41035
I20260809 00:52:30.315636 14939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:41035 every 8 connection(s)
I20260809 00:52:30.317116 14725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:52:30.322600 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 14725
I20260809 00:52:30.322816 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:52:30.323278 10116 external_mini_cluster.cc:1428] Setting key b230d276604e71609aa5654bc491f2fc
Aug 09 00:52:30 dist-test-slave-j2vl krb5kdc[14144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236750, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:30.339741 14944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:30.320114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:45495 (local address 127.9.225.62:37475)
0809 00:52:30.320654 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:30.320659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:30.321593 (+ 934us) server_negotiation.cc:407] Connection header received
0809 00:52:30.323797 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:30.323803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:30.323907 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:30.324033 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:30.326132 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:30.327072 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:30.328776 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:30.329230 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:30.329336 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:30.333103 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:30.333109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:30.333112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:30.333149 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:30.336043 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:30.337019 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:30.337023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:30.337025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:30.337155 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:30.337756 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:30.337761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:30.337763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:30.338077 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:30.338270 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:30.339087 (+ 817us) server_negotiation.cc:299] Negotiation successful
0809 00:52:30.339349 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":282,"threads_started":1}
I20260809 00:52:30.340924 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:30.320546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:37475 (local address 127.9.225.3:45495)
0809 00:52:30.321392 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:30.321456 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:30.323479 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:30.324373 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:30.324386 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:30.325041 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:30.325932 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:30.325961 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:30.327237 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:30.327242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:30.328614 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:30.328629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:30.329516 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:30.329522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:30.329777 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:30.330477 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:30.330502 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:30.332847 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:30.336261 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:30.336271 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:30.336289 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:30.336858 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:30.337280 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:30.337288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:30.337294 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:30.337598 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:30.338223 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:30.338234 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:30.338646 (+ 412us) client_negotiation.cc:769] Sending connection context
0809 00:52:30.338954 (+ 308us) client_negotiation.cc:240] Negotiation successful
0809 00:52:30.339586 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":306,"threads_started":1}
I20260809 00:52:30.342578 14940 heartbeater.cc:347] Connected to a master server at 127.9.225.62:37475
I20260809 00:52:30.342976 14940 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:30.343825 14940 heartbeater.cc:511] Master 127.9.225.62:37475 requested a full tablet report, sending...
I20260809 00:52:30.345564 14220 ts_manager.cc:194] Registered new tserver with Master: 2a4954bd74d842e9b74a187504c46a4b (127.9.225.3:41035)
I20260809 00:52:30.346395 14220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:45495
I20260809 00:52:30.352083 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 00:52:30.398303 14944 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:52:30.394651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.9.225.62:37475)
0809 00:52:30.394993 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:30.395000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:30.395562 (+ 562us) server_negotiation.cc:407] Connection header received
0809 00:52:30.395785 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:30.395789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:30.395887 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:30.396008 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:30.397902 (+ 1894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32870: BlockingRecv error: recv got EOF from 127.0.0.1:32870 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
I20260809 00:52:30.402711 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 14314
W20260809 00:52:30.406796 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 14314: Not authorized: Client connection negotiation failed: client connection to 127.9.225.1:41711: server requires authentication, but client does not have Kerberos credentials available
I20260809 00:52:30.407099 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 14314
I20260809 00:52:30.441666 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 14520
W20260809 00:52:30.444895 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 14520: Not authorized: Client connection negotiation failed: client connection to 127.9.225.2:39571: server requires authentication, but client does not have Kerberos credentials available
I20260809 00:52:30.445084 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 14520
I20260809 00:52:30.494756 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 14725
W20260809 00:52:30.498410 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 14725: Not authorized: Client connection negotiation failed: client connection to 127.9.225.3:41035: server requires authentication, but client does not have Kerberos credentials available
I20260809 00:52:30.498623 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 14725
I20260809 00:52:30.530977 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 14163
I20260809 00:52:30.532305 14269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:30.641601 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 14163
2026-08-09T00:52:30Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4569 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:52:30 dist-test-slave-j2vl krb5kdc[14978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:52:30 dist-test-slave-j2vl krb5kdc[14978](info): set up 2 sockets
Aug 09 00:52:30 dist-test-slave-j2vl krb5kdc[14978](info): commencing operation
krb5kdc: starting...
W20260809 00:52:32.008981 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.317s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:52:32 dist-test-slave-j2vl krb5kdc[14978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:52:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:52:32.275861 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:33731
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:35467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:33731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:32.655311 15002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:32.655673 15002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:32.655779 15002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:32.663353 15002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:32.663484 15002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:32.663535 15002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:32.663587 15002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:32.663640 15002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:52:32.676640 15002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:35467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:33731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:33731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:32.678233 15002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:32.679723 15002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:32.693215 15016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:32.693333 15013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:32.693244 15012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:32.694141 15002 server_base.cc:1034] running on GCE node
I20260809 00:52:32.695191 15002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:32.697005 15002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:32.698339 15002 hybrid_clock.cc:648] HybridClock initialized: now 1786236752698261 us; error 105 us; skew 500 ppm
I20260809 00:52:32.698848 15002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:32 dist-test-slave-j2vl krb5kdc[14978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:32.702692 15002 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:52:32.704715 15002 webserver.cc:460] Webserver started at http://127.9.225.62:41451/ using document root <none> and password file <none>
I20260809 00:52:32.705648 15002 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:32.705775 15002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:32.706207 15002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:32.709849 15002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "843f22a204384712b88d5a2c6beeab75"
format_stamp: "Formatted at 2026-08-09 00:52:32 on dist-test-slave-j2vl"
server_key: "7913d09a5306893d701a5ccb9d621929"
server_key_iv: "0a27127f035a8e541b75ba41d45aa8f8"
server_key_version: "encryptionkey@0"
I20260809 00:52:32.710842 15002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "843f22a204384712b88d5a2c6beeab75"
format_stamp: "Formatted at 2026-08-09 00:52:32 on dist-test-slave-j2vl"
server_key: "7913d09a5306893d701a5ccb9d621929"
server_key_iv: "0a27127f035a8e541b75ba41d45aa8f8"
server_key_version: "encryptionkey@0"
I20260809 00:52:32.717929 15002 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260809 00:52:32.722178 15027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:32.723905 15002 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260809 00:52:32.724161 15002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "843f22a204384712b88d5a2c6beeab75"
format_stamp: "Formatted at 2026-08-09 00:52:32 on dist-test-slave-j2vl"
server_key: "7913d09a5306893d701a5ccb9d621929"
server_key_iv: "0a27127f035a8e541b75ba41d45aa8f8"
server_key_version: "encryptionkey@0"
I20260809 00:52:32.724422 15002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:32.748773 15002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:32.753700 15002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:32.754122 15002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:32.775398 15002 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:33731
I20260809 00:52:32.775429 15105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:33731 every 8 connection(s)
I20260809 00:52:32.777107 15002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:32.781739 15108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:32.785179 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 15002
I20260809 00:52:32.785454 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:32.786304 10116 external_mini_cluster.cc:1428] Setting key 5339fab0792ca3175a3076e1b7483303
I20260809 00:52:32.797633 15108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75: Bootstrap starting.
I20260809 00:52:32.802462 15108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:32.804591 15108 log.cc:824] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75: Log is configured to *not* fsync() on all Append() calls
Aug 09 00:52:32 dist-test-slave-j2vl krb5kdc[14978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:32.808269 15108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75: No bootstrap required, opened a new log
I20260809 00:52:32.815507 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:32.789202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49162 (local address 127.9.225.62:33731)
0809 00:52:32.790482 (+ 1280us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:32.790495 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:32.790544 (+ 49us) server_negotiation.cc:407] Connection header received
0809 00:52:32.792481 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:32.792514 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:32.793046 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:32.793559 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:32.795654 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:32.797583 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:32.799631 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:32.800458 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:32.800631 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:32.806887 (+ 6256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:32.806897 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:32.806918 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:32.806961 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:32.810490 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:32.811381 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:32.811390 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:32.811397 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:32.811726 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:32.812162 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:32.812168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:32.812171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:32.812819 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:32.813197 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:32.813785 (+ 588us) server_negotiation.cc:299] Negotiation successful
0809 00:52:32.814286 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":637,"threads_started":1}
I20260809 00:52:32.817625 15108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843f22a204384712b88d5a2c6beeab75" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33731 } }
I20260809 00:52:32.818202 15108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:32.818349 15108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 843f22a204384712b88d5a2c6beeab75, State: Initialized, Role: FOLLOWER
I20260809 00:52:32.819295 15108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843f22a204384712b88d5a2c6beeab75" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33731 } }
I20260809 00:52:32.819661 15108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:32.819828 15108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:32.820048 15108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:32.821417 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:52:32.823287 15108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843f22a204384712b88d5a2c6beeab75" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33731 } }
I20260809 00:52:32.824105 15108 leader_election.cc:302] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 843f22a204384712b88d5a2c6beeab75; no voters:
I20260809 00:52:32.825028 15108 leader_election.cc:288] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:32.825254 15116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:32.826603 15116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [term 1 LEADER]: Becoming Leader. State: Replica: 843f22a204384712b88d5a2c6beeab75, State: Running, Role: LEADER
I20260809 00:52:32.827509 15116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843f22a204384712b88d5a2c6beeab75" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33731 } }
I20260809 00:52:32.828307 15108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:32.832787 15117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "843f22a204384712b88d5a2c6beeab75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843f22a204384712b88d5a2c6beeab75" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33731 } } }
I20260809 00:52:32.833262 15117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:32.834057 15120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 843f22a204384712b88d5a2c6beeab75. Latest consensus state: current_term: 1 leader_uuid: "843f22a204384712b88d5a2c6beeab75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "843f22a204384712b88d5a2c6beeab75" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33731 } } }
I20260809 00:52:32.834421 15120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:32.840812 15131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:32.853732 15131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:32.871438 15131 catalog_manager.cc:1347] Generated new cluster ID: c84a323b71ab4c49b42633c90b2dc1ba
I20260809 00:52:32.871594 15131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:32.901230 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:32.890175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49170 (local address 127.9.225.62:33731)
0809 00:52:32.890456 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:32.890467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:32.890987 (+ 520us) server_negotiation.cc:407] Connection header received
0809 00:52:32.891217 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:32.891224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:32.891312 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:32.891425 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:32.892988 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:32.894212 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:32.896332 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:32.897027 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:32.897160 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:32.898684 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:32.898695 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:32.898701 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:32.899852 (+ 1151us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:32.899964 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:32.900001 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:32.900129 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:32.900355 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 00:52:32.900616 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 00:52:32.920621 15131 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:32.922222 15131 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:32.938354 15131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 843f22a204384712b88d5a2c6beeab75: Generated new TSK 0
I20260809 00:52:32.939644 15131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:52:32.952809 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 15002
I20260809 00:52:32.954319 15100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:33.054062 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 15002
2026-08-09T00:52:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2416 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:52:33 dist-test-slave-j2vl krb5kdc[15166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:52:33 dist-test-slave-j2vl krb5kdc[15166](info): set up 2 sockets
Aug 09 00:52:33 dist-test-slave-j2vl krb5kdc[15166](info): commencing operation
krb5kdc: starting...
W20260809 00:52:34.433909 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.325s user 0.007s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:52:34 dist-test-slave-j2vl krb5kdc[15166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:52:34.705454 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:46461
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:46461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:35.090874 15183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:35.091254 15183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:35.091418 15183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:35.099601 15183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:35.099762 15183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:35.099859 15183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:35.099977 15183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:35.100046 15183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:52:35.110898 15183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:46461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:46461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:35.112511 15183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:35.114133 15183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:35.127969 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:35.128024 15194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:35.127939 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:35.128984 15183 server_base.cc:1034] running on GCE node
I20260809 00:52:35.130036 15183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:35.131786 15183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:35.133100 15183 hybrid_clock.cc:648] HybridClock initialized: now 1786236755133040 us; error 51 us; skew 500 ppm
I20260809 00:52:35.133615 15183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:35 dist-test-slave-j2vl krb5kdc[15166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:35.137946 15183 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:52:35.140359 15183 webserver.cc:460] Webserver started at http://127.9.225.62:37679/ using document root <none> and password file <none>
I20260809 00:52:35.141327 15183 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:35.141597 15183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:35.141994 15183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:35.145879 15183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "f34762c3615e4e86a626219652d5b5f9"
format_stamp: "Formatted at 2026-08-09 00:52:35 on dist-test-slave-j2vl"
server_key: "a09528b86426fffecd492c44d9d233c7"
server_key_iv: "5e51189003c4f04f450a9188bffe0b58"
server_key_version: "encryptionkey@0"
I20260809 00:52:35.146843 15183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "f34762c3615e4e86a626219652d5b5f9"
format_stamp: "Formatted at 2026-08-09 00:52:35 on dist-test-slave-j2vl"
server_key: "a09528b86426fffecd492c44d9d233c7"
server_key_iv: "5e51189003c4f04f450a9188bffe0b58"
server_key_version: "encryptionkey@0"
I20260809 00:52:35.153280 15183 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 00:52:35.157640 15208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:35.159312 15183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 00:52:35.159550 15183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "f34762c3615e4e86a626219652d5b5f9"
format_stamp: "Formatted at 2026-08-09 00:52:35 on dist-test-slave-j2vl"
server_key: "a09528b86426fffecd492c44d9d233c7"
server_key_iv: "5e51189003c4f04f450a9188bffe0b58"
server_key_version: "encryptionkey@0"
I20260809 00:52:35.159790 15183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:35.181082 15183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:35.186360 15183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:35.186709 15183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:35.208300 15183 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:46461
I20260809 00:52:35.208424 15297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:46461 every 8 connection(s)
I20260809 00:52:35.210008 15183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:35.214679 15300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:35.214709 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 15183
I20260809 00:52:35.214972 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:35.215693 10116 external_mini_cluster.cc:1428] Setting key 8abf02924e0cd5d4e763066ef3f819ed
I20260809 00:52:35.232622 15300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9: Bootstrap starting.
Aug 09 00:52:35 dist-test-slave-j2vl krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:35.237067 15300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:35.238471 15300 log.cc:824] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:35.242266 15300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9: No bootstrap required, opened a new log
I20260809 00:52:35.244603 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:35.217992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.9.225.62:46461)
0809 00:52:35.219151 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:35.219164 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:35.219213 (+ 49us) server_negotiation.cc:407] Connection header received
0809 00:52:35.222010 (+ 2797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:35.222047 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:35.222639 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:35.223077 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:35.225403 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:35.226849 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:35.229015 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:35.229994 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:35.230173 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:35.235902 (+ 5729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:35.235913 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:35.235931 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:35.235981 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:35.239680 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:35.240639 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:35.240648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:35.240656 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:35.240863 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:35.241319 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:35.241332 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:35.241335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:35.242025 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:35.242433 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:35.242918 (+ 485us) server_negotiation.cc:299] Negotiation successful
0809 00:52:35.243422 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":639,"threads_started":1}
I20260809 00:52:35.251986 15300 raft_consensus.cc:348] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34762c3615e4e86a626219652d5b5f9" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 46461 } }
I20260809 00:52:35.252616 15300 raft_consensus.cc:374] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:35.252744 15300 raft_consensus.cc:729] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34762c3615e4e86a626219652d5b5f9, State: Initialized, Role: FOLLOWER
I20260809 00:52:35.253654 15300 consensus_queue.cc:260] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34762c3615e4e86a626219652d5b5f9" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 46461 } }
I20260809 00:52:35.253914 15300 raft_consensus.cc:388] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:35.254029 15300 raft_consensus.cc:482] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:35.254169 15300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:35.256994 15300 raft_consensus.cc:504] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34762c3615e4e86a626219652d5b5f9" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 46461 } }
I20260809 00:52:35.257728 15300 leader_election.cc:302] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f34762c3615e4e86a626219652d5b5f9; no voters:
I20260809 00:52:35.258447 15300 leader_election.cc:288] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:35.258741 15311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:35.259526 15311 raft_consensus.cc:686] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [term 1 LEADER]: Becoming Leader. State: Replica: f34762c3615e4e86a626219652d5b5f9, State: Running, Role: LEADER
I20260809 00:52:35.260180 15311 consensus_queue.cc:237] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34762c3615e4e86a626219652d5b5f9" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 46461 } }
I20260809 00:52:35.261024 15300 sys_catalog.cc:564] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:35.263622 15313 sys_catalog.cc:455] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f34762c3615e4e86a626219652d5b5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34762c3615e4e86a626219652d5b5f9" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 46461 } } }
I20260809 00:52:35.263720 15314 sys_catalog.cc:455] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f34762c3615e4e86a626219652d5b5f9. Latest consensus state: current_term: 1 leader_uuid: "f34762c3615e4e86a626219652d5b5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f34762c3615e4e86a626219652d5b5f9" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 46461 } } }
I20260809 00:52:35.264142 15313 sys_catalog.cc:458] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:35.264205 15314 sys_catalog.cc:458] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:35.270004 15317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:35.276337 15317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:35.288337 15317 catalog_manager.cc:1347] Generated new cluster ID: 97289ff7f6b245029d7465c3f385e070
I20260809 00:52:35.288532 15317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:35.307298 15317 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:35.308861 15317 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:35.317593 15317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f34762c3615e4e86a626219652d5b5f9: Generated new TSK 0
I20260809 00:52:35.318727 15317 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:52:35.397258 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:46461
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:35.805058 15339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:35.805452 15339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:35.805562 15339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:35.813501 15339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:35.813642 15339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:35.813763 15339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:52:35.824643 15339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:46461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:35.826252 15339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:35.827728 15339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:35.842906 15347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:35.842927 15351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:35.842959 15353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:35.844195 15339 server_base.cc:1034] running on GCE node
I20260809 00:52:35.844991 15339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:35.846288 15339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:35.847558 15339 hybrid_clock.cc:648] HybridClock initialized: now 1786236755847482 us; error 68 us; skew 500 ppm
I20260809 00:52:35.847963 15339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:35 dist-test-slave-j2vl krb5kdc[15166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:35.855501 15339 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:52:35.857484 15339 webserver.cc:460] Webserver started at http://127.9.225.1:40527/ using document root <none> and password file <none>
I20260809 00:52:35.858346 15339 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:35.858522 15339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:35.858901 15339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:35.862877 15339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "3ea24f64e4644c499f045aecef824b4d"
format_stamp: "Formatted at 2026-08-09 00:52:35 on dist-test-slave-j2vl"
server_key: "ca752708849a68b56629b85a6e0c6fb7"
server_key_iv: "83df0a47aa6ae1a8a14daeec468c834b"
server_key_version: "encryptionkey@0"
I20260809 00:52:35.864068 15339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "3ea24f64e4644c499f045aecef824b4d"
format_stamp: "Formatted at 2026-08-09 00:52:35 on dist-test-slave-j2vl"
server_key: "ca752708849a68b56629b85a6e0c6fb7"
server_key_iv: "83df0a47aa6ae1a8a14daeec468c834b"
server_key_version: "encryptionkey@0"
I20260809 00:52:35.871037 15339 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 00:52:35.875883 15365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:35.877782 15339 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260809 00:52:35.878053 15339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "3ea24f64e4644c499f045aecef824b4d"
format_stamp: "Formatted at 2026-08-09 00:52:35 on dist-test-slave-j2vl"
server_key: "ca752708849a68b56629b85a6e0c6fb7"
server_key_iv: "83df0a47aa6ae1a8a14daeec468c834b"
server_key_version: "encryptionkey@0"
I20260809 00:52:35.878312 15339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:35.894826 15339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:35.899806 15339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:35.900195 15339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:35.901376 15339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:35.903028 15339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:35.903154 15339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:35.903322 15339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:35.903405 15339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:35.940753 15339 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:43777
I20260809 00:52:35.940879 15555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:43777 every 8 connection(s)
I20260809 00:52:35.942309 15339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:52:35.945914 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 15339
I20260809 00:52:35.946134 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:52:35.947080 10116 external_mini_cluster.cc:1428] Setting key e05f0d22aeb0429f4c0392704426459d
Aug 09 00:52:35 dist-test-slave-j2vl krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236755, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:35.969470 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:35.945335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:36447 (local address 127.9.225.62:46461)
0809 00:52:35.945892 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:35.945900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:35.947334 (+ 1434us) server_negotiation.cc:407] Connection header received
0809 00:52:35.949967 (+ 2633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:35.949972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:35.950046 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:35.950139 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:35.952743 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:35.953613 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:35.956460 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:35.956894 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:35.957007 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:35.961946 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:35.961954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:35.961959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:35.962002 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:35.964987 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:35.966310 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:35.966315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:35.966317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:35.966484 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:35.967171 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:35.967175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:35.967177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:35.967513 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:35.967637 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:35.968840 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0809 00:52:35.969066 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":244,"threads_started":1}
I20260809 00:52:35.971285 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:35.946035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:46461 (local address 127.9.225.1:36447)
0809 00:52:35.947117 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:35.947180 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:35.949576 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:35.950491 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:35.950506 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:35.951455 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:35.952476 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:35.952510 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:35.953830 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:35.953838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:35.956217 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:35.956241 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:35.957169 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:35.957178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:35.957594 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:35.958472 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:35.958511 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:35.961519 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:35.965234 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:35.965251 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:35.965278 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:35.966128 (+ 850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:35.966652 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:35.966660 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:35.966666 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:35.967018 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:35.967659 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:35.967671 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:35.968305 (+ 634us) client_negotiation.cc:769] Sending connection context
0809 00:52:35.968683 (+ 378us) client_negotiation.cc:240] Negotiation successful
0809 00:52:35.969456 (+ 773us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":377,"threads_started":1}
I20260809 00:52:35.973412 15556 heartbeater.cc:347] Connected to a master server at 127.9.225.62:46461
I20260809 00:52:35.973850 15556 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
I20260809 00:52:35.975054 15556 heartbeater.cc:511] Master 127.9.225.62:46461 requested a full tablet report, sending...
I20260809 00:52:35.977706 15242 ts_manager.cc:194] Registered new tserver with Master: 3ea24f64e4644c499f045aecef824b4d (127.9.225.1:43777)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
I20260809 00:52:35.980091 15242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:36447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:52:36.009557 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:46461
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:36.391630 15564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:36.392064 15564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:36.392252 15564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:36.400158 15564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:36.400372 15564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:36.400578 15564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:52:36.411221 15564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:46461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:36.412832 15564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:36.414470 15564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:36.429464 15575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:36.429500 15573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:36.429524 15571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:36.430581 15564 server_base.cc:1034] running on GCE node
I20260809 00:52:36.431360 15564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:36.432549 15564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:36.433800 15564 hybrid_clock.cc:648] HybridClock initialized: now 1786236756433747 us; error 50 us; skew 500 ppm
I20260809 00:52:36.434182 15564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:36 dist-test-slave-j2vl krb5kdc[15166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236756, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:36.438004 15564 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:52:36.439669 15564 webserver.cc:460] Webserver started at http://127.9.225.2:38437/ using document root <none> and password file <none>
I20260809 00:52:36.440440 15564 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:36.440557 15564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:36.440834 15564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:36.444480 15564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "227ac382cac540ee811ca496dda1c642"
format_stamp: "Formatted at 2026-08-09 00:52:36 on dist-test-slave-j2vl"
server_key: "c63cea0c1e7b323f339b57fcc1ac6c83"
server_key_iv: "ea5de86901b976b77ab17402211de67a"
server_key_version: "encryptionkey@0"
I20260809 00:52:36.445531 15564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "227ac382cac540ee811ca496dda1c642"
format_stamp: "Formatted at 2026-08-09 00:52:36 on dist-test-slave-j2vl"
server_key: "c63cea0c1e7b323f339b57fcc1ac6c83"
server_key_iv: "ea5de86901b976b77ab17402211de67a"
server_key_version: "encryptionkey@0"
I20260809 00:52:36.451699 15564 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:52:36.455989 15584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:36.457764 15564 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260809 00:52:36.457990 15564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "227ac382cac540ee811ca496dda1c642"
format_stamp: "Formatted at 2026-08-09 00:52:36 on dist-test-slave-j2vl"
server_key: "c63cea0c1e7b323f339b57fcc1ac6c83"
server_key_iv: "ea5de86901b976b77ab17402211de67a"
server_key_version: "encryptionkey@0"
I20260809 00:52:36.458292 15564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:36.485908 15564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:36.491034 15564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:36.491398 15564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:36.492545 15564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:36.494227 15564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:36.494351 15564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:36.494520 15564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:36.494597 15564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:36.531848 15564 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:46539
I20260809 00:52:36.531903 15772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:46539 every 8 connection(s)
I20260809 00:52:36.533388 15564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:52:36.537947 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 15564
I20260809 00:52:36.538159 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:52:36.538769 10116 external_mini_cluster.cc:1428] Setting key ec16c0263451181519b17dd6eb8646a9
Aug 09 00:52:36 dist-test-slave-j2vl krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236756, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:36.557435 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:36.536485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:46755 (local address 127.9.225.62:46461)
0809 00:52:36.537011 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:36.537020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:36.538459 (+ 1439us) server_negotiation.cc:407] Connection header received
0809 00:52:36.541052 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:36.541061 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:36.541196 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:36.541330 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:36.543277 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:36.544040 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:36.545894 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:36.546349 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:36.546499 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:36.550721 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:36.550728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:36.550732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:36.550773 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:36.553805 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:36.554767 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:36.554772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:36.554774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:36.554913 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:36.555435 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:36.555439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:36.555441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:36.555746 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:36.555927 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:36.556733 (+ 806us) server_negotiation.cc:299] Negotiation successful
0809 00:52:36.556988 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":233,"threads_started":1}
I20260809 00:52:36.558660 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:36.537123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:46461 (local address 127.9.225.2:46755)
0809 00:52:36.538243 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:36.538313 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:36.540606 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:36.541734 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:36.541746 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:36.542384 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:36.543063 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:36.543085 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:36.544243 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:36.544249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:36.545713 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:36.545729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:36.546623 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:36.546631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:36.546891 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:36.547679 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:36.547710 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:36.550391 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:36.554005 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:36.554017 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:36.554044 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:36.554589 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:36.555042 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:36.555048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:36.555052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:36.555284 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:36.555901 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:36.555911 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:36.556322 (+ 411us) client_negotiation.cc:769] Sending connection context
0809 00:52:36.556628 (+ 306us) client_negotiation.cc:240] Negotiation successful
0809 00:52:36.557276 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":470,"threads_started":1}
I20260809 00:52:36.560542 15775 heartbeater.cc:347] Connected to a master server at 127.9.225.62:46461
I20260809 00:52:36.560932 15775 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:36.561964 15775 heartbeater.cc:511] Master 127.9.225.62:46461 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
I20260809 00:52:36.564074 15242 ts_manager.cc:194] Registered new tserver with Master: 227ac382cac540ee811ca496dda1c642 (127.9.225.2:46539)
I20260809 00:52:36.565114 15242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:46755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:52:36.598230 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:46461
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:36.978986 15784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:36.979346 15784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:36.979450 15784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 00:52:36.984915 15556 heartbeater.cc:503] Master 127.9.225.62:46461 was elected leader, sending a full tablet report...
W20260809 00:52:36.988258 15784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:36.988399 15784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:36.988607 15784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:52:36.999380 15784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:46461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:37.000958 15784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:37.002475 15784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:37.017876 15798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:37.017829 15800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:37.017783 15803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:37.018631 15784 server_base.cc:1034] running on GCE node
I20260809 00:52:37.019393 15784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:37.020473 15784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:37.021687 15784 hybrid_clock.cc:648] HybridClock initialized: now 1786236757021632 us; error 51 us; skew 500 ppm
I20260809 00:52:37.022056 15784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:37 dist-test-slave-j2vl krb5kdc[15166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:37.025887 15784 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:52:37.027673 15784 webserver.cc:460] Webserver started at http://127.9.225.3:35867/ using document root <none> and password file <none>
I20260809 00:52:37.028487 15784 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:37.028650 15784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:37.028990 15784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:37.032318 15784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "1a5597b30fc348db870d17dad569ea18"
format_stamp: "Formatted at 2026-08-09 00:52:37 on dist-test-slave-j2vl"
server_key: "3d4c96b058c2f693b04f14338422f953"
server_key_iv: "023464d7c0458ee829db9ecf24fb0ac6"
server_key_version: "encryptionkey@0"
I20260809 00:52:37.033257 15784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "1a5597b30fc348db870d17dad569ea18"
format_stamp: "Formatted at 2026-08-09 00:52:37 on dist-test-slave-j2vl"
server_key: "3d4c96b058c2f693b04f14338422f953"
server_key_iv: "023464d7c0458ee829db9ecf24fb0ac6"
server_key_version: "encryptionkey@0"
I20260809 00:52:37.039873 15784 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:52:37.044015 15811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:37.045751 15784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 00:52:37.045969 15784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "1a5597b30fc348db870d17dad569ea18"
format_stamp: "Formatted at 2026-08-09 00:52:37 on dist-test-slave-j2vl"
server_key: "3d4c96b058c2f693b04f14338422f953"
server_key_iv: "023464d7c0458ee829db9ecf24fb0ac6"
server_key_version: "encryptionkey@0"
I20260809 00:52:37.046209 15784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:37.060659 15784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:37.066042 15784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:37.066336 15784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:37.067341 15784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:37.068831 15784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:37.068913 15784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:37.069031 15784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:37.069080 15784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:37.106297 15784 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:42931
I20260809 00:52:37.106318 15986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:42931 every 8 connection(s)
I20260809 00:52:37.107815 15784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:52:37.116711 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 15784
I20260809 00:52:37.116950 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:52:37.117513 10116 external_mini_cluster.cc:1428] Setting key 1766bc9a72e8dcb99a653e19ae08d379
Aug 09 00:52:37 dist-test-slave-j2vl krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236757, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:37.131333 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:37.110876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:37935 (local address 127.9.225.62:46461)
0809 00:52:37.111404 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:37.111409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:37.112513 (+ 1104us) server_negotiation.cc:407] Connection header received
0809 00:52:37.114649 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:37.114655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:37.114795 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:37.114928 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:37.116794 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:37.117858 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:37.120068 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:37.120519 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:37.120672 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:37.124936 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:37.124942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:37.124946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:37.124987 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:37.127959 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:37.128869 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:37.128874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:37.128876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:37.129013 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:37.129547 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:37.129551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:37.129553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:37.129872 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:37.130013 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:37.130765 (+ 752us) server_negotiation.cc:299] Negotiation successful
0809 00:52:37.130975 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
I20260809 00:52:37.132560 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:37.111282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:46461 (local address 127.9.225.3:37935)
0809 00:52:37.112333 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:37.112377 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:37.114316 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:37.115225 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:37.115235 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:37.115860 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:37.116566 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:37.116588 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:37.118056 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:37.118061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:37.119891 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:37.119913 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:37.120802 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:37.120810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:37.121077 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:37.121921 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:37.121948 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:37.124616 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:37.128159 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:37.128169 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:37.128186 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:37.128716 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:37.129124 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:37.129127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:37.129130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:37.129403 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:37.129993 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:37.130000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:37.130381 (+ 381us) client_negotiation.cc:769] Sending connection context
0809 00:52:37.130627 (+ 246us) client_negotiation.cc:240] Negotiation successful
0809 00:52:37.131088 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":357,"threads_started":1}
I20260809 00:52:37.134357 15987 heartbeater.cc:347] Connected to a master server at 127.9.225.62:46461
I20260809 00:52:37.134724 15987 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:37.135569 15987 heartbeater.cc:511] Master 127.9.225.62:46461 requested a full tablet report, sending...
I20260809 00:52:37.137307 15241 ts_manager.cc:194] Registered new tserver with Master: 1a5597b30fc348db870d17dad569ea18 (127.9.225.3:42931)
I20260809 00:52:37.138315 15241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:37935
I20260809 00:52:37.146908 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 00:52:37 dist-test-slave-j2vl krb5kdc[15166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236757, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 00:52:37 dist-test-slave-j2vl krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236757, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:37.209764 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:37.194117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.9.225.62:46461)
0809 00:52:37.194390 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:37.194397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:37.194711 (+ 314us) server_negotiation.cc:407] Connection header received
0809 00:52:37.195008 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:37.195013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:37.195089 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:37.195186 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:37.196637 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:37.197566 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:37.198890 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:37.199309 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:37.199456 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:37.203807 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:37.203814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:37.203819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:37.203880 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:37.206729 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:37.207586 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:37.207591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:37.207593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:37.207733 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:37.208262 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:37.208266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:37.208268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:37.208674 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:37.208890 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:37.209236 (+ 346us) server_negotiation.cc:299] Negotiation successful
0809 00:52:37.209499 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260809 00:52:37.210453 15241 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58410
I20260809 00:52:37.212954 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 15339
Aug 09 00:52:37 dist-test-slave-j2vl krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236757, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:37.234396 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:37.214223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.9.225.1:43777)
0809 00:52:37.215003 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:37.215014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:37.217661 (+ 2647us) server_negotiation.cc:407] Connection header received
0809 00:52:37.217894 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:37.217903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:37.218181 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:37.218373 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:37.219842 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:37.220899 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:37.222297 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:37.222974 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:37.223138 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:37.227647 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:37.227664 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:37.227685 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:37.227739 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:37.231061 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:37.231959 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:37.231968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:37.231970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:37.232121 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:37.232560 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:37.232564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:37.232566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:37.232973 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:37.233319 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:37.233698 (+ 379us) server_negotiation.cc:299] Negotiation successful
0809 00:52:37.234019 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":356,"threads_started":1}
W20260809 00:52:37.235865 15548 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37712
W20260809 00:52:37.237300 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 15339: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:52:37.237599 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 15339
I20260809 00:52:37.274156 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 15564
Aug 09 00:52:37 dist-test-slave-j2vl krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236757, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:52:37.293241 16009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:37.275351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.9.225.2:46539)
0809 00:52:37.276051 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:37.276059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:37.276090 (+ 31us) server_negotiation.cc:407] Connection header received
0809 00:52:37.276163 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:37.276170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:37.276407 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:37.276563 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:37.278055 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:37.279146 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:37.280546 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:37.281206 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:37.281330 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:37.286055 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:37.286075 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:37.286095 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:37.286140 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:37.289900 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:37.290837 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:37.290847 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:37.290849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:37.291018 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:37.291415 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:37.291419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:37.291421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:37.291882 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:37.292139 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:37.292531 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 00:52:37.292796 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":395,"threads_started":1}
W20260809 00:52:37.294767 15766 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46794
W20260809 00:52:37.296207 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 15564: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:52:37.296412 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 15564
I20260809 00:52:37.329480 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 15784
Aug 09 00:52:37 dist-test-slave-j2vl krb5kdc[15166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236757, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:52:37.347738 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:37.330514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.9.225.3:42931)
0809 00:52:37.331160 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:37.331169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:37.331201 (+ 32us) server_negotiation.cc:407] Connection header received
0809 00:52:37.331264 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:37.331288 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:37.331528 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:37.331670 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:37.333031 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:37.334358 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:37.335655 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:37.336255 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:37.336369 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:37.340934 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:37.340951 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:37.340971 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:37.341015 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:37.344448 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:37.345330 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:37.345500 (+ 170us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:37.345507 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:37.345671 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:37.346153 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:37.346158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:37.346160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:37.346600 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:37.346869 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:37.347198 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 00:52:37.347435 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":330,"threads_started":1}
W20260809 00:52:37.349242 15979 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42750
W20260809 00:52:37.350725 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 15784: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:52:37.350909 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 15784
I20260809 00:52:37.386092 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 15183
I20260809 00:52:37.387465 15288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:37.498171 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 15183
2026-08-09T00:52:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4442 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:52:37 dist-test-slave-j2vl krb5kdc[16015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:52:37 dist-test-slave-j2vl krb5kdc[16015](info): set up 2 sockets
Aug 09 00:52:37 dist-test-slave-j2vl krb5kdc[16015](info): commencing operation
krb5kdc: starting...
W20260809 00:52:38.859395 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.309s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:52:39 dist-test-slave-j2vl krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:52:39.125758 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:33819
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:33819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:39.505716 16034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:39.506095 16034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:39.506268 16034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:39.514246 16034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:39.514705 16034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:39.514834 16034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:39.514947 16034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:39.515219 16034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:52:39.526316 16034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:33819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:33819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:39.527874 16034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:39.529515 16034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:39.542969 16041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:39.543046 16043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:39.543013 16046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:39.544402 16034 server_base.cc:1034] running on GCE node
I20260809 00:52:39.545307 16034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:39.547003 16034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:39.548234 16034 hybrid_clock.cc:648] HybridClock initialized: now 1786236759548157 us; error 71 us; skew 500 ppm
I20260809 00:52:39.548653 16034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:39 dist-test-slave-j2vl krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236759, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:39.552395 16034 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:52:39.554272 16034 webserver.cc:460] Webserver started at http://127.9.225.62:38451/ using document root <none> and password file <none>
I20260809 00:52:39.555109 16034 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:39.555253 16034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:39.555632 16034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:39.559190 16034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "83a7d042e1bb485db999be3cb3289044"
format_stamp: "Formatted at 2026-08-09 00:52:39 on dist-test-slave-j2vl"
server_key: "0ff9a77e1840423cb8bbf4bba750a448"
server_key_iv: "8dde294e32b0ef97762d98da466dfbd7"
server_key_version: "encryptionkey@0"
I20260809 00:52:39.560129 16034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "83a7d042e1bb485db999be3cb3289044"
format_stamp: "Formatted at 2026-08-09 00:52:39 on dist-test-slave-j2vl"
server_key: "0ff9a77e1840423cb8bbf4bba750a448"
server_key_iv: "8dde294e32b0ef97762d98da466dfbd7"
server_key_version: "encryptionkey@0"
I20260809 00:52:39.566417 16034 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 00:52:39.570564 16056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:39.572114 16034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260809 00:52:39.572340 16034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "83a7d042e1bb485db999be3cb3289044"
format_stamp: "Formatted at 2026-08-09 00:52:39 on dist-test-slave-j2vl"
server_key: "0ff9a77e1840423cb8bbf4bba750a448"
server_key_iv: "8dde294e32b0ef97762d98da466dfbd7"
server_key_version: "encryptionkey@0"
I20260809 00:52:39.572571 16034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:39.597873 16034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:39.603226 16034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:39.603578 16034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:39.624326 16034 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:33819
I20260809 00:52:39.624349 16135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:33819 every 8 connection(s)
I20260809 00:52:39.626026 16034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:39.630745 16136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:39.633894 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 16034
I20260809 00:52:39.634184 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:39.635248 10116 external_mini_cluster.cc:1428] Setting key 25d38d54326a68169291de918d7a8e62
I20260809 00:52:39.646514 16136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044: Bootstrap starting.
I20260809 00:52:39.650863 16136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:39.652053 16136 log.cc:824] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044: Log is configured to *not* fsync() on all Append() calls
Aug 09 00:52:39 dist-test-slave-j2vl krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:39.655993 16136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044: No bootstrap required, opened a new log
I20260809 00:52:39.663060 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:39.637944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.9.225.62:33819)
0809 00:52:39.638915 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:39.638935 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:39.638999 (+ 64us) server_negotiation.cc:407] Connection header received
0809 00:52:39.641663 (+ 2664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:39.641714 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:39.642422 (+ 708us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:39.642995 (+ 573us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:39.644598 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:39.646728 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:39.648084 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:39.649323 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:39.649573 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:39.654381 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:39.654391 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:39.654413 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:39.654460 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:39.657928 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:39.659005 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:39.659017 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:39.659028 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:39.659232 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:39.659696 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:39.659700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:39.659702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:39.660354 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:39.660679 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:39.661252 (+ 573us) server_negotiation.cc:299] Negotiation successful
0809 00:52:39.661861 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":414,"threads_started":1}
I20260809 00:52:39.665665 16136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a7d042e1bb485db999be3cb3289044" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33819 } }
I20260809 00:52:39.666143 16136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:39.666280 16136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a7d042e1bb485db999be3cb3289044, State: Initialized, Role: FOLLOWER
I20260809 00:52:39.666895 16136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a7d042e1bb485db999be3cb3289044" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33819 } }
I20260809 00:52:39.667171 16136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:39.667321 16136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:39.667503 16136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:39.670073 16136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a7d042e1bb485db999be3cb3289044" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33819 } }
I20260809 00:52:39.670661 16136 leader_election.cc:302] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83a7d042e1bb485db999be3cb3289044; no voters:
I20260809 00:52:39.671322 16136 leader_election.cc:288] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:39.671482 16143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:39.672504 16143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [term 1 LEADER]: Becoming Leader. State: Replica: 83a7d042e1bb485db999be3cb3289044, State: Running, Role: LEADER
I20260809 00:52:39.673559 16143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a7d042e1bb485db999be3cb3289044" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33819 } }
I20260809 00:52:39.673892 16136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:39.676447 16145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83a7d042e1bb485db999be3cb3289044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a7d042e1bb485db999be3cb3289044" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33819 } } }
I20260809 00:52:39.676865 16145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:39.677455 16146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83a7d042e1bb485db999be3cb3289044. Latest consensus state: current_term: 1 leader_uuid: "83a7d042e1bb485db999be3cb3289044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a7d042e1bb485db999be3cb3289044" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33819 } } }
I20260809 00:52:39.677824 16146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:39.678313 16154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:39.691707 16154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:39.702798 16154 catalog_manager.cc:1347] Generated new cluster ID: 7fc914cdb55047b3b7dd90b7b84265f7
I20260809 00:52:39.702946 16154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:39.733259 16154 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:39.735514 16154 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:39.746306 16154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83a7d042e1bb485db999be3cb3289044: Generated new TSK 0
I20260809 00:52:39.747552 16154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:52:39.821645 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33819
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:40.204783 16177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:40.205142 16177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:40.205315 16177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:40.213502 16177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:40.213657 16177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:40.213853 16177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:52:40.224611 16177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:40.226235 16177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:40.227771 16177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:40.243137 16183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:40.243209 16186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:40.243094 16184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:40.249745 16177 server_base.cc:1034] running on GCE node
I20260809 00:52:40.250622 16177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:40.251948 16177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:40.253245 16177 hybrid_clock.cc:648] HybridClock initialized: now 1786236760253156 us; error 78 us; skew 500 ppm
I20260809 00:52:40.253688 16177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:40 dist-test-slave-j2vl krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236760, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:40.257529 16177 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:52:40.259517 16177 webserver.cc:460] Webserver started at http://127.9.225.1:46079/ using document root <none> and password file <none>
I20260809 00:52:40.260406 16177 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:40.260573 16177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:40.260939 16177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:40.264443 16177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "6bbe66891e6e464a9e7848a54c5228de"
format_stamp: "Formatted at 2026-08-09 00:52:40 on dist-test-slave-j2vl"
server_key: "f38a610743e857b6a5511c3ccc65a6f8"
server_key_iv: "7d3a0856f14b16032beaf2688e237163"
server_key_version: "encryptionkey@0"
I20260809 00:52:40.265492 16177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "6bbe66891e6e464a9e7848a54c5228de"
format_stamp: "Formatted at 2026-08-09 00:52:40 on dist-test-slave-j2vl"
server_key: "f38a610743e857b6a5511c3ccc65a6f8"
server_key_iv: "7d3a0856f14b16032beaf2688e237163"
server_key_version: "encryptionkey@0"
I20260809 00:52:40.271512 16177 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260809 00:52:40.275950 16200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:40.277941 16177 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:52:40.278160 16177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "6bbe66891e6e464a9e7848a54c5228de"
format_stamp: "Formatted at 2026-08-09 00:52:40 on dist-test-slave-j2vl"
server_key: "f38a610743e857b6a5511c3ccc65a6f8"
server_key_iv: "7d3a0856f14b16032beaf2688e237163"
server_key_version: "encryptionkey@0"
I20260809 00:52:40.278417 16177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:40.304862 16177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:40.309960 16177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:40.310292 16177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:40.311424 16177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:40.313002 16177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:40.313114 16177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:40.313269 16177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:40.313344 16177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:40.349390 16177 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:45783
I20260809 00:52:40.349485 16361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:45783 every 8 connection(s)
I20260809 00:52:40.350857 16177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:52:40.359407 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 16177
I20260809 00:52:40.359690 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:52:40.360517 10116 external_mini_cluster.cc:1428] Setting key d9a04b2d69c27d9c8f7b3616e64f8cd2
Aug 09 00:52:40 dist-test-slave-j2vl krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236760, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:40.376122 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:40.353818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:50711 (local address 127.9.225.62:33819)
0809 00:52:40.354412 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:40.354417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:40.355411 (+ 994us) server_negotiation.cc:407] Connection header received
0809 00:52:40.357592 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:40.357598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:40.357708 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:40.357863 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:40.360355 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:40.361299 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:40.363381 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:40.363813 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:40.363972 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:40.368561 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:40.368568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:40.368571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:40.368613 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:40.371527 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:40.372872 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:40.372878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:40.372880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:40.373032 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:40.373664 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:40.373669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:40.373672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:40.374107 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:40.374243 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:40.375514 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0809 00:52:40.375836 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":328,"threads_started":1}
I20260809 00:52:40.377692 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:40.354356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33819 (local address 127.9.225.1:50711)
0809 00:52:40.355199 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:40.355260 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:40.357171 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:40.358185 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:40.358195 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:40.358960 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:40.360037 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:40.360075 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:40.361585 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:40.361589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:40.363197 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:40.363213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:40.364096 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:40.364106 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:40.364382 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:40.365186 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:40.365213 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:40.368213 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:40.371770 (+ 3557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:40.371786 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:40.371821 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:40.372687 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:40.373158 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:40.373166 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:40.373173 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:40.373509 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:40.374259 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:40.374271 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:40.374852 (+ 581us) client_negotiation.cc:769] Sending connection context
0809 00:52:40.375227 (+ 375us) client_negotiation.cc:240] Negotiation successful
0809 00:52:40.375965 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":308,"threads_started":1}
I20260809 00:52:40.379727 16362 heartbeater.cc:347] Connected to a master server at 127.9.225.62:33819
I20260809 00:52:40.380167 16362 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:40.381014 16362 heartbeater.cc:511] Master 127.9.225.62:33819 requested a full tablet report, sending...
I20260809 00:52:40.383594 16082 ts_manager.cc:194] Registered new tserver with Master: 6bbe66891e6e464a9e7848a54c5228de (127.9.225.1:45783)
I20260809 00:52:40.385182 16082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:50711
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:52:40.421279 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33819
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:40.818579 16370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:40.818936 16370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:40.819047 16370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:40.826675 16370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:40.826835 16370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:40.827092 16370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:52:40.837808 16370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:40.839275 16370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:40.840729 16370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:40.856313 16384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:40.856273 16379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:40.856353 16380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:40.857200 16370 server_base.cc:1034] running on GCE node
I20260809 00:52:40.857977 16370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:40.859127 16370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:40.860388 16370 hybrid_clock.cc:648] HybridClock initialized: now 1786236760860290 us; error 70 us; skew 500 ppm
I20260809 00:52:40.860735 16370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:40 dist-test-slave-j2vl krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236760, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:40.875463 16370 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:52:40.877609 16370 webserver.cc:460] Webserver started at http://127.9.225.2:45461/ using document root <none> and password file <none>
I20260809 00:52:40.878496 16370 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:40.878667 16370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:40.879036 16370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:40.882828 16370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "b14b12a65edf479f84354006992f9440"
format_stamp: "Formatted at 2026-08-09 00:52:40 on dist-test-slave-j2vl"
server_key: "98ba983a27f22441255ce65e24a4fa1e"
server_key_iv: "6d88bab06c8263e7a6a9148329127136"
server_key_version: "encryptionkey@0"
I20260809 00:52:40.883759 16370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "b14b12a65edf479f84354006992f9440"
format_stamp: "Formatted at 2026-08-09 00:52:40 on dist-test-slave-j2vl"
server_key: "98ba983a27f22441255ce65e24a4fa1e"
server_key_iv: "6d88bab06c8263e7a6a9148329127136"
server_key_version: "encryptionkey@0"
I20260809 00:52:40.890069 16370 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260809 00:52:40.895146 16395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:40.896924 16370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:52:40.897149 16370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "b14b12a65edf479f84354006992f9440"
format_stamp: "Formatted at 2026-08-09 00:52:40 on dist-test-slave-j2vl"
server_key: "98ba983a27f22441255ce65e24a4fa1e"
server_key_iv: "6d88bab06c8263e7a6a9148329127136"
server_key_version: "encryptionkey@0"
I20260809 00:52:40.897470 16370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:40.934777 16370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:40.939654 16370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:40.940007 16370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:40.941169 16370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:40.942763 16370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:40.942848 16370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:40.943037 16370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:40.943089 16370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:40.983094 16370 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:46431
I20260809 00:52:40.983158 16569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:46431 every 8 connection(s)
I20260809 00:52:40.984654 16370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:52:40.990957 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 16370
I20260809 00:52:40.991202 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:52:40.991777 10116 external_mini_cluster.cc:1428] Setting key b290b2100dd80e6b0f76cc740e8ed034
Aug 09 00:52:40 dist-test-slave-j2vl krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236760, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:41.012362 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:40.987690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:50189 (local address 127.9.225.62:33819)
0809 00:52:40.988260 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:40.988265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:40.989537 (+ 1272us) server_negotiation.cc:407] Connection header received
0809 00:52:40.992202 (+ 2665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:40.992208 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:40.992292 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:40.992405 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:40.994596 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:40.995388 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:40.998337 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:40.998819 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:40.998959 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:41.004081 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:41.004091 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:41.004098 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:41.004155 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:41.008165 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.009192 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.009201 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.009208 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.009447 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.009965 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.009973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.009980 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.010417 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:41.010586 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:41.011517 (+ 931us) server_negotiation.cc:299] Negotiation successful
0809 00:52:41.011843 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":309,"threads_started":1}
I20260809 00:52:41.013322 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:40.988420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33819 (local address 127.9.225.2:50189)
0809 00:52:40.989274 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:40.989319 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:40.991792 (+ 2473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:40.992692 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:40.992701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:40.993525 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:40.994378 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:40.994407 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:40.995586 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:40.995595 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:40.998093 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:40.998119 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:40.999121 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:40.999130 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:40.999515 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:41.000412 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:41.000460 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:41.003697 (+ 3237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:41.008402 (+ 4705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:41.008412 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:41.008432 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:41.009016 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:41.009576 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:41.009579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:41.009582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:41.009823 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:41.010587 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:41.010595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:41.011052 (+ 457us) client_negotiation.cc:769] Sending connection context
0809 00:52:41.011331 (+ 279us) client_negotiation.cc:240] Negotiation successful
0809 00:52:41.011842 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":378,"threads_started":1}
I20260809 00:52:41.015220 16570 heartbeater.cc:347] Connected to a master server at 127.9.225.62:33819
I20260809 00:52:41.015653 16570 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:41.016779 16570 heartbeater.cc:511] Master 127.9.225.62:33819 requested a full tablet report, sending...
I20260809 00:52:41.019009 16082 ts_manager.cc:194] Registered new tserver with Master: b14b12a65edf479f84354006992f9440 (127.9.225.2:46431)
I20260809 00:52:41.020350 16082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:50189
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:52:41.055236 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33819
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:45919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:52:41.388571 16362 heartbeater.cc:503] Master 127.9.225.62:33819 was elected leader, sending a full tablet report...
W20260809 00:52:41.434376 16579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:41.434749 16579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:41.434962 16579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:41.443040 16579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:41.443195 16579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:41.443398 16579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:52:41.454577 16579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:45919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:41.456166 16579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:41.457717 16579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:41.472791 16591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:41.472879 16589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:41.472827 16587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:41.473821 16579 server_base.cc:1034] running on GCE node
I20260809 00:52:41.474632 16579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:41.475769 16579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:41.476994 16579 hybrid_clock.cc:648] HybridClock initialized: now 1786236761476900 us; error 78 us; skew 500 ppm
I20260809 00:52:41.477491 16579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:41 dist-test-slave-j2vl krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236761, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:41.481138 16579 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:52:41.483042 16579 webserver.cc:460] Webserver started at http://127.9.225.3:43835/ using document root <none> and password file <none>
I20260809 00:52:41.483920 16579 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:41.484071 16579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:41.484385 16579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:41.487954 16579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "8222a77a5d0e47e6b34bdbfd20e6acbf"
format_stamp: "Formatted at 2026-08-09 00:52:41 on dist-test-slave-j2vl"
server_key: "0f4bae2b42ee8ee34a463b0385e99042"
server_key_iv: "62122f3355c60ca7b321b7ada03c2820"
server_key_version: "encryptionkey@0"
I20260809 00:52:41.488885 16579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "8222a77a5d0e47e6b34bdbfd20e6acbf"
format_stamp: "Formatted at 2026-08-09 00:52:41 on dist-test-slave-j2vl"
server_key: "0f4bae2b42ee8ee34a463b0385e99042"
server_key_iv: "62122f3355c60ca7b321b7ada03c2820"
server_key_version: "encryptionkey@0"
I20260809 00:52:41.495302 16579 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.007s
I20260809 00:52:41.499619 16604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:41.501291 16579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 00:52:41.501627 16579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "8222a77a5d0e47e6b34bdbfd20e6acbf"
format_stamp: "Formatted at 2026-08-09 00:52:41 on dist-test-slave-j2vl"
server_key: "0f4bae2b42ee8ee34a463b0385e99042"
server_key_iv: "62122f3355c60ca7b321b7ada03c2820"
server_key_version: "encryptionkey@0"
I20260809 00:52:41.501889 16579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:41.527164 16579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:41.532588 16579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:41.532940 16579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:41.534255 16579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:41.535778 16579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:41.535859 16579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:41.535990 16579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:41.536048 16579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:41.576483 16579 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:37161
I20260809 00:52:41.576653 16782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:37161 every 8 connection(s)
I20260809 00:52:41.578088 16579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:52:41.583750 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 16579
I20260809 00:52:41.584013 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:52:41.584511 10116 external_mini_cluster.cc:1428] Setting key 2561840168c4a4c9606c1129afc3ba68
Aug 09 00:52:41 dist-test-slave-j2vl krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236761, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:41.603556 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:41.581216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:60083 (local address 127.9.225.62:33819)
0809 00:52:41.581883 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:41.581890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:41.582759 (+ 869us) server_negotiation.cc:407] Connection header received
0809 00:52:41.585484 (+ 2725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:41.585491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:41.585612 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:41.585763 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:41.588507 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.589449 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.591281 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.591748 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.591891 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:41.596217 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:41.596223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:41.596227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:41.596270 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:41.599515 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.600487 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.600494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.600497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.600648 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.601244 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.601273 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.601277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.601761 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:41.601949 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:41.602823 (+ 874us) server_negotiation.cc:299] Negotiation successful
0809 00:52:41.603094 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":393,"threads_started":1}
I20260809 00:52:41.604730 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:41.581734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33819 (local address 127.9.225.3:60083)
0809 00:52:41.582574 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:41.582617 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:41.584950 (+ 2333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:41.586195 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:41.586211 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:41.587126 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:41.588229 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:41.588266 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.589645 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.589650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:41.591094 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:41.591112 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.592116 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.592124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:41.592373 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:41.593182 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:41.593212 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:41.595902 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:41.599738 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:41.599748 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:41.599768 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:41.600343 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:41.600808 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:41.600812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:41.600815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:41.601098 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:41.601950 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:41.601962 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:41.602402 (+ 440us) client_negotiation.cc:769] Sending connection context
0809 00:52:41.602719 (+ 317us) client_negotiation.cc:240] Negotiation successful
0809 00:52:41.603416 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":300,"threads_started":1}
I20260809 00:52:41.606406 16783 heartbeater.cc:347] Connected to a master server at 127.9.225.62:33819
I20260809 00:52:41.606786 16783 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:41.607805 16783 heartbeater.cc:511] Master 127.9.225.62:33819 requested a full tablet report, sending...
I20260809 00:52:41.609908 16082 ts_manager.cc:194] Registered new tserver with Master: 8222a77a5d0e47e6b34bdbfd20e6acbf (127.9.225.3:37161)
I20260809 00:52:41.610810 16082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:60083
I20260809 00:52:41.614162 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 00:52:41 dist-test-slave-j2vl krb5kdc[16015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 00:52:41 dist-test-slave-j2vl krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:41.679984 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:41.661898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.9.225.1:45783)
0809 00:52:41.662609 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:41.662619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:41.662651 (+ 32us) server_negotiation.cc:407] Connection header received
0809 00:52:41.662724 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:41.662731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:41.662963 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:41.663129 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:41.664764 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.665998 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.667585 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.668238 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.668385 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:41.672797 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:41.672816 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:41.672837 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:41.672891 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:41.676415 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.677446 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.677454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.677457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.677646 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.678155 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.678158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.678160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.678614 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:41.678891 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:41.679283 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 00:52:41.679536 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":388,"threads_started":1}
Aug 09 00:52:41 dist-test-slave-j2vl krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:41.701828 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:41.685990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35890 (local address 127.9.225.62:33819)
0809 00:52:41.686258 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:41.686265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:41.686523 (+ 258us) server_negotiation.cc:407] Connection header received
0809 00:52:41.686750 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:41.686756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:41.686871 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:41.686986 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:41.688490 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.689249 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.690601 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.691068 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.691242 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:41.695890 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:41.695897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:41.695901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:41.695942 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:41.698856 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.699634 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.699638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.699641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.699775 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.700113 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.700116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.700118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.700470 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:41.700669 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:41.701238 (+ 569us) server_negotiation.cc:299] Negotiation successful
0809 00:52:41.701535 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260809 00:52:41.702505 16082 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35890
I20260809 00:52:41.703969 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 16177
I20260809 00:52:41.718370 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:41.705224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48504 (local address 127.9.225.1:45783)
0809 00:52:41.705482 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:41.705489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:41.705653 (+ 164us) server_negotiation.cc:407] Connection header received
0809 00:52:41.705852 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:41.705857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:41.705962 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:41.706081 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:41.707503 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.708274 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.709586 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.710024 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.710138 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:41.712609 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:41.712615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:41.712618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:41.712660 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:41.715517 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.716260 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.716265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.716267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.716415 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.716869 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.716872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.716881 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.717143 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:41.717322 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:41.717823 (+ 501us) server_negotiation.cc:299] Negotiation successful
0809 00:52:41.718019 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 00:52:41.719177 16354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:41.807343 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 16177
I20260809 00:52:41.837801 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 16370
Aug 09 00:52:41 dist-test-slave-j2vl krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:52:41.856323 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:41.838925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.9.225.2:46431)
0809 00:52:41.839535 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:41.839543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:41.839572 (+ 29us) server_negotiation.cc:407] Connection header received
0809 00:52:41.839640 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:41.839647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:41.839892 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:41.840054 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:41.841270 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.842480 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.843924 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.844572 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.844684 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:41.849648 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:41.849675 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:41.849695 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:41.849753 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:41.852898 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.853897 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.853907 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.853909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.854061 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.854489 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.854494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.854501 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.854909 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:41.855245 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:41.855587 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 00:52:41.855902 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":308,"threads_started":1}
I20260809 00:52:41.857827 16564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:41.946308 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 16370
I20260809 00:52:41.974438 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 16579
Aug 09 00:52:41 dist-test-slave-j2vl krb5kdc[16015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:52:41.993582 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:41.975635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.9.225.3:37161)
0809 00:52:41.976325 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:41.976334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:41.976376 (+ 42us) server_negotiation.cc:407] Connection header received
0809 00:52:41.976441 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:41.976448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:41.976691 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:41.976888 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:41.978431 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.979664 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.981051 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:41.981886 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:41.982009 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:41.986739 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:41.986760 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:41.986783 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:41.986844 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:41.990239 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.991145 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.991154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.991156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.991344 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:41.991757 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:41.991762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:41.991764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:41.992196 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:41.992472 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:41.992759 (+ 287us) server_negotiation.cc:299] Negotiation successful
0809 00:52:41.993012 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":341,"threads_started":1}
I20260809 00:52:41.995169 16776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:42.083801 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 16579
I20260809 00:52:42.112357 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 16034
I20260809 00:52:42.113730 16129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:42.225036 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 16034
2026-08-09T00:52:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4729 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:52:42 dist-test-slave-j2vl krb5kdc[16819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:52:42 dist-test-slave-j2vl krb5kdc[16819](info): set up 2 sockets
Aug 09 00:52:42 dist-test-slave-j2vl krb5kdc[16819](info): commencing operation
krb5kdc: starting...
W20260809 00:52:43.625447 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.345s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:52:43 dist-test-slave-j2vl krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:52:43.902551 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39131
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:39131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:44.279094 16837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:44.279438 16837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:44.279543 16837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:44.287269 16837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:44.287415 16837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:44.287463 16837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:44.287528 16837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:44.287577 16837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:52:44.299209 16837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:39131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:44.300709 16837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:44.302202 16837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:44.315282 16850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:44.315209 16847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:44.315209 16846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:44.316121 16837 server_base.cc:1034] running on GCE node
I20260809 00:52:44.317137 16837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:44.318888 16837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:44.320132 16837 hybrid_clock.cc:648] HybridClock initialized: now 1786236764320076 us; error 43 us; skew 500 ppm
I20260809 00:52:44.320585 16837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:44 dist-test-slave-j2vl krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236764, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:44.324353 16837 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:52:44.325886 16837 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:44.325997 16837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:44.326326 16837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:44.329829 16837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "7f89afe7936f499998430eec4bb0e959"
format_stamp: "Formatted at 2026-08-09 00:52:44 on dist-test-slave-j2vl"
server_key: "09774f761a78583fa3760a1baeb92209"
server_key_iv: "fcc197a3a05b21b41f83ed00520401d1"
server_key_version: "encryptionkey@0"
I20260809 00:52:44.330709 16837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "7f89afe7936f499998430eec4bb0e959"
format_stamp: "Formatted at 2026-08-09 00:52:44 on dist-test-slave-j2vl"
server_key: "09774f761a78583fa3760a1baeb92209"
server_key_iv: "fcc197a3a05b21b41f83ed00520401d1"
server_key_version: "encryptionkey@0"
I20260809 00:52:44.337458 16837 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 00:52:44.342085 16859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:44.343801 16837 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:52:44.344063 16837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "7f89afe7936f499998430eec4bb0e959"
format_stamp: "Formatted at 2026-08-09 00:52:44 on dist-test-slave-j2vl"
server_key: "09774f761a78583fa3760a1baeb92209"
server_key_iv: "fcc197a3a05b21b41f83ed00520401d1"
server_key_version: "encryptionkey@0"
I20260809 00:52:44.344362 16837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:44.372512 16837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:44.377638 16837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:44.377915 16837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:44.400465 16837 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:39131
I20260809 00:52:44.400480 16935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:39131 every 8 connection(s)
I20260809 00:52:44.401605 16837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:44.406008 16938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:44.411358 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 16837
I20260809 00:52:44.411629 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:44.412467 10116 external_mini_cluster.cc:1428] Setting key 235d655c30527215895c203184930823
I20260809 00:52:44.420033 16938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959: Bootstrap starting.
I20260809 00:52:44.424467 16938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:44.425956 16938 log.cc:824] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959: Log is configured to *not* fsync() on all Append() calls
Aug 09 00:52:44 dist-test-slave-j2vl krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:44.430051 16938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959: No bootstrap required, opened a new log
I20260809 00:52:44.439908 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:44.415004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.9.225.62:39131)
0809 00:52:44.416007 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:44.416021 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:44.416074 (+ 53us) server_negotiation.cc:407] Connection header received
0809 00:52:44.418236 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:44.418270 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:44.418829 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:44.419246 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:44.421546 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:44.423021 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:44.425007 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:44.425898 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:44.426066 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:44.431407 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:44.431418 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:44.431438 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:44.431485 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:44.434962 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:44.435998 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:44.436006 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:44.436014 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:44.436228 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:44.436695 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:44.436699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:44.436701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:44.437480 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:44.437730 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:44.438243 (+ 513us) server_negotiation.cc:299] Negotiation successful
0809 00:52:44.438731 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":484,"threads_started":1}
I20260809 00:52:44.439721 16938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f89afe7936f499998430eec4bb0e959" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39131 } }
I20260809 00:52:44.440156 16938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:44.440305 16938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f89afe7936f499998430eec4bb0e959, State: Initialized, Role: FOLLOWER
I20260809 00:52:44.440989 16938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f89afe7936f499998430eec4bb0e959" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39131 } }
I20260809 00:52:44.441205 16938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:44.441365 16938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:44.441569 16938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:44.444590 16938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f89afe7936f499998430eec4bb0e959" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39131 } }
I20260809 00:52:44.445191 16938 leader_election.cc:302] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f89afe7936f499998430eec4bb0e959; no voters:
I20260809 00:52:44.445886 16938 leader_election.cc:288] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:44.446089 16946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:44.446816 16946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [term 1 LEADER]: Becoming Leader. State: Replica: 7f89afe7936f499998430eec4bb0e959, State: Running, Role: LEADER
I20260809 00:52:44.447438 16946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f89afe7936f499998430eec4bb0e959" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39131 } }
I20260809 00:52:44.448559 16938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:44.451375 16948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f89afe7936f499998430eec4bb0e959. Latest consensus state: current_term: 1 leader_uuid: "7f89afe7936f499998430eec4bb0e959" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f89afe7936f499998430eec4bb0e959" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39131 } } }
I20260809 00:52:44.452026 16948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:44.451172 16947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f89afe7936f499998430eec4bb0e959" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f89afe7936f499998430eec4bb0e959" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39131 } } }
I20260809 00:52:44.453995 16947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:44.455217 16958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:44.467658 16958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:44.479573 16958 catalog_manager.cc:1347] Generated new cluster ID: 105ee573a7c844d6ae0a97d4e01515a5
I20260809 00:52:44.479739 16958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:44.501757 16958 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:44.504251 16958 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:44.518501 16958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959: Generated new TSK 0
I20260809 00:52:44.519855 16958 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:52:44.593324 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:39131
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:44.973959 16986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:44.974323 16986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:44.974494 16986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:44.982483 16986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:44.982630 16986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:44.982833 16986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:52:44.994040 16986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:39131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:44.995712 16986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:44.997249 16986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:45.012195 16991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:45.012269 16993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:45.012269 16995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:45.013545 16986 server_base.cc:1034] running on GCE node
I20260809 00:52:45.014310 16986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:45.015465 16986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:45.016721 16986 hybrid_clock.cc:648] HybridClock initialized: now 1786236765016673 us; error 60 us; skew 500 ppm
I20260809 00:52:45.017158 16986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:45 dist-test-slave-j2vl krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:45.021118 16986 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:52:45.022867 16986 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:45.023026 16986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:45.023389 16986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:45.026775 16986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "6c13ddfa9d1547dfb367b2dd2b01da09"
format_stamp: "Formatted at 2026-08-09 00:52:45 on dist-test-slave-j2vl"
server_key: "34c97de2ebbc7055eadba15bb13c78ac"
server_key_iv: "136cc5b479a66578de89e8fa75afbe6e"
server_key_version: "encryptionkey@0"
I20260809 00:52:45.027705 16986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "6c13ddfa9d1547dfb367b2dd2b01da09"
format_stamp: "Formatted at 2026-08-09 00:52:45 on dist-test-slave-j2vl"
server_key: "34c97de2ebbc7055eadba15bb13c78ac"
server_key_iv: "136cc5b479a66578de89e8fa75afbe6e"
server_key_version: "encryptionkey@0"
I20260809 00:52:45.034020 16986 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 00:52:45.038439 17002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:45.040073 16986 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:52:45.040318 16986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "6c13ddfa9d1547dfb367b2dd2b01da09"
format_stamp: "Formatted at 2026-08-09 00:52:45 on dist-test-slave-j2vl"
server_key: "34c97de2ebbc7055eadba15bb13c78ac"
server_key_iv: "136cc5b479a66578de89e8fa75afbe6e"
server_key_version: "encryptionkey@0"
I20260809 00:52:45.040601 16986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:45.061398 16986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:45.066406 16986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:45.066751 16986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:45.067878 16986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:45.069509 16986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:45.069638 16986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:45.069789 16986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:45.069877 16986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:45.107280 16986 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:34095
I20260809 00:52:45.107343 17173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:34095 every 8 connection(s)
I20260809 00:52:45.108302 16986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:52:45.112324 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 16986
I20260809 00:52:45.112588 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:52:45.113509 10116 external_mini_cluster.cc:1428] Setting key 1ee357c8c1965a7fc0f18b719b165286
Aug 09 00:52:45 dist-test-slave-j2vl krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:45.131666 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:45.111441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:44433 (local address 127.9.225.62:39131)
0809 00:52:45.112029 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:45.112038 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:45.113321 (+ 1283us) server_negotiation.cc:407] Connection header received
0809 00:52:45.115585 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:45.115591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:45.115699 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:45.115830 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:45.117584 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:45.118357 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:45.119999 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:45.120427 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:45.120568 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:45.124686 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:45.124698 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:45.124705 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:45.124756 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:45.127567 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:45.128723 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:45.128728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:45.128730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:45.128875 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:45.129558 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:45.129563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:45.129565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:45.129893 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:45.130033 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:45.131150 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0809 00:52:45.131379 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":244,"threads_started":1}
I20260809 00:52:45.133908 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:45.112106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39131 (local address 127.9.225.1:44433)
0809 00:52:45.113082 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:45.113141 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:45.115277 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:45.116099 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:45.116108 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:45.116693 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:45.117394 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:45.117423 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:45.118526 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:45.118537 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:45.119831 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:45.119844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:45.120674 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:45.120683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:45.120922 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:45.121722 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:45.121747 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:45.124367 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:45.127765 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:45.127780 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:45.127806 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:45.128564 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:45.128991 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:45.128998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:45.129004 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:45.129281 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:45.130053 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:45.130066 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:45.130615 (+ 549us) client_negotiation.cc:769] Sending connection context
0809 00:52:45.130984 (+ 369us) client_negotiation.cc:240] Negotiation successful
0809 00:52:45.131689 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":404,"threads_started":1}
I20260809 00:52:45.135968 17175 heartbeater.cc:347] Connected to a master server at 127.9.225.62:39131
I20260809 00:52:45.136368 17175 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:45.137204 17175 heartbeater.cc:511] Master 127.9.225.62:39131 requested a full tablet report, sending...
I20260809 00:52:45.139886 16881 ts_manager.cc:194] Registered new tserver with Master: 6c13ddfa9d1547dfb367b2dd2b01da09 (127.9.225.1:34095)
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
I20260809 00:52:45.141592 16881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:44433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:52:45.177151 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:39131
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:45.560660 17184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:45.561043 17184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:45.561226 17184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:45.569566 17184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:45.569720 17184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:45.569908 17184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:52:45.580922 17184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:39131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:45.582670 17184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:45.584200 17184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:45.600922 17196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:45.601192 17201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:45.600922 17199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:45.602896 17184 server_base.cc:1034] running on GCE node
I20260809 00:52:45.603971 17184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:45.605767 17184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:45.607115 17184 hybrid_clock.cc:648] HybridClock initialized: now 1786236765607029 us; error 71 us; skew 500 ppm
I20260809 00:52:45.607555 17184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:45 dist-test-slave-j2vl krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:45.611567 17184 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:52:45.613222 17184 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:45.613417 17184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:45.613817 17184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:45.617259 17184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "b0913aaf5aa245eab06ba7eb4ef63e65"
format_stamp: "Formatted at 2026-08-09 00:52:45 on dist-test-slave-j2vl"
server_key: "c08184f36b18f40e0c5cd4953469b88b"
server_key_iv: "f4513c784a8bfd11c5c062ba3cde9c7f"
server_key_version: "encryptionkey@0"
I20260809 00:52:45.618300 17184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "b0913aaf5aa245eab06ba7eb4ef63e65"
format_stamp: "Formatted at 2026-08-09 00:52:45 on dist-test-slave-j2vl"
server_key: "c08184f36b18f40e0c5cd4953469b88b"
server_key_iv: "f4513c784a8bfd11c5c062ba3cde9c7f"
server_key_version: "encryptionkey@0"
I20260809 00:52:45.625129 17184 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 00:52:45.630599 17211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:45.632400 17184 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:52:45.632697 17184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "b0913aaf5aa245eab06ba7eb4ef63e65"
format_stamp: "Formatted at 2026-08-09 00:52:45 on dist-test-slave-j2vl"
server_key: "c08184f36b18f40e0c5cd4953469b88b"
server_key_iv: "f4513c784a8bfd11c5c062ba3cde9c7f"
server_key_version: "encryptionkey@0"
I20260809 00:52:45.633005 17184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:45.647929 17184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:45.653748 17184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:45.654129 17184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:45.655555 17184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:45.657438 17184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:45.657553 17184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 00:52:45.657729 17184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:45.657823 17184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:45.695916 17184 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:41265
I20260809 00:52:45.695951 17383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:41265 every 8 connection(s)
I20260809 00:52:45.696978 17184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:52:45.705829 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 17184
I20260809 00:52:45.706069 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:52:45.706549 10116 external_mini_cluster.cc:1428] Setting key eaabaed94132de242676febf1e4392a1
Aug 09 00:52:45 dist-test-slave-j2vl krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236765, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:45.725063 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:45.699962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:50071 (local address 127.9.225.62:39131)
0809 00:52:45.700491 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:45.700496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:45.702049 (+ 1553us) server_negotiation.cc:407] Connection header received
0809 00:52:45.704129 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:45.704134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:45.704247 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:45.704376 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:45.706430 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:45.707252 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:45.709241 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:45.710092 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:45.710597 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:45.715578 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:45.715602 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:45.715608 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:45.715679 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:45.720038 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:45.721473 (+ 1435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:45.721482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:45.721488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:45.721696 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:45.722521 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:45.722530 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:45.722536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:45.723010 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:45.723188 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:45.724217 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0809 00:52:45.724542 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":281,"threads_started":1}
I20260809 00:52:45.726012 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:45.700562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39131 (local address 127.9.225.2:50071)
0809 00:52:45.701787 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:45.701842 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:45.703757 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:45.704665 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:45.704674 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:45.705333 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:45.706201 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:45.706227 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:45.707452 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:45.707457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:45.709035 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:45.709052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:45.710635 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:45.710642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:45.710915 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:45.711960 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:45.712003 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:45.715138 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:45.720300 (+ 5162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:45.720315 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:45.720343 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:45.721226 (+ 883us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:45.721917 (+ 691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:45.721922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:45.721926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:45.722261 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:45.723254 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:45.723263 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:45.723741 (+ 478us) client_negotiation.cc:769] Sending connection context
0809 00:52:45.724049 (+ 308us) client_negotiation.cc:240] Negotiation successful
0809 00:52:45.724564 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":465,"threads_started":1}
I20260809 00:52:45.727967 17384 heartbeater.cc:347] Connected to a master server at 127.9.225.62:39131
I20260809 00:52:45.728346 17384 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:45.729115 17384 heartbeater.cc:511] Master 127.9.225.62:39131 requested a full tablet report, sending...
I20260809 00:52:45.731021 16881 ts_manager.cc:194] Registered new tserver with Master: b0913aaf5aa245eab06ba7eb4ef63e65 (127.9.225.2:41265)
I20260809 00:52:45.732455 16881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:50071
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:52:45.773578 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:39131
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:52:46.145218 17175 heartbeater.cc:503] Master 127.9.225.62:39131 was elected leader, sending a full tablet report...
W20260809 00:52:46.155184 17393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:46.155530 17393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:46.155637 17393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:46.163291 17393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:46.163439 17393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:46.163563 17393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:52:46.174074 17393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:39131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:46.175611 17393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:46.177003 17393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:46.191582 17401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:46.191612 17404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:46.191650 17400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:46.192795 17393 server_base.cc:1034] running on GCE node
I20260809 00:52:46.193710 17393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:46.195004 17393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:46.196296 17393 hybrid_clock.cc:648] HybridClock initialized: now 1786236766196215 us; error 78 us; skew 500 ppm
I20260809 00:52:46.196692 17393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:46 dist-test-slave-j2vl krb5kdc[16819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236766, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:46.205077 17393 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:52:46.206645 17393 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:46.206807 17393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:46.207192 17393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:46.210889 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "e0e206a157804986b89c15b0f4ce080a"
format_stamp: "Formatted at 2026-08-09 00:52:46 on dist-test-slave-j2vl"
server_key: "45d3813512fcec0813038661a764cbde"
server_key_iv: "7cc46b0a34ead7a91c34107cb2408a8f"
server_key_version: "encryptionkey@0"
I20260809 00:52:46.211889 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "e0e206a157804986b89c15b0f4ce080a"
format_stamp: "Formatted at 2026-08-09 00:52:46 on dist-test-slave-j2vl"
server_key: "45d3813512fcec0813038661a764cbde"
server_key_iv: "7cc46b0a34ead7a91c34107cb2408a8f"
server_key_version: "encryptionkey@0"
I20260809 00:52:46.217927 17393 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260809 00:52:46.222034 17412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:46.224110 17393 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 00:52:46.224370 17393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "e0e206a157804986b89c15b0f4ce080a"
format_stamp: "Formatted at 2026-08-09 00:52:46 on dist-test-slave-j2vl"
server_key: "45d3813512fcec0813038661a764cbde"
server_key_iv: "7cc46b0a34ead7a91c34107cb2408a8f"
server_key_version: "encryptionkey@0"
I20260809 00:52:46.224658 17393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:46.251804 17393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:46.257411 17393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:46.257733 17393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:46.258852 17393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:46.260531 17393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:46.260635 17393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:46.260802 17393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:46.260887 17393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:46.296350 17393 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:37513
I20260809 00:52:46.296402 17589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:37513 every 8 connection(s)
I20260809 00:52:46.297449 17393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:52:46.302649 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 17393
I20260809 00:52:46.302914 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:52:46.303359 10116 external_mini_cluster.cc:1428] Setting key 6ff9ab1f38d6c6223929ac4b8d4ee1f4
Aug 09 00:52:46 dist-test-slave-j2vl krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236766, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:46.321225 17593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.300515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:52199 (local address 127.9.225.62:39131)
0809 00:52:46.301090 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:46.301098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:46.302346 (+ 1248us) server_negotiation.cc:407] Connection header received
0809 00:52:46.304694 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:46.304700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:46.304839 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:46.304974 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:46.307228 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.308077 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.309902 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.310341 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.310447 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.314497 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:46.314503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:46.314507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:46.314549 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:46.317393 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:46.318526 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:46.318532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:46.318536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:46.318733 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:46.319292 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:46.319297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:46.319300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:46.319681 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:46.319859 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:46.320723 (+ 864us) server_negotiation.cc:299] Negotiation successful
0809 00:52:46.320944 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":282,"threads_started":1}
I20260809 00:52:46.322402 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.301186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39131 (local address 127.9.225.3:52199)
0809 00:52:46.302104 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:46.302154 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:46.304325 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:46.305377 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:46.305393 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:46.306171 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:46.307011 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:46.307042 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.308247 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.308251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:46.309715 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:46.309729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.310579 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.310585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:46.310832 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.311573 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:46.311601 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:46.314207 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:46.317590 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:46.317600 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:46.317619 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:46.318375 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:46.318859 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:46.318863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:46.318865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:46.319161 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:46.319865 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:46.319874 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:46.320317 (+ 443us) client_negotiation.cc:769] Sending connection context
0809 00:52:46.320604 (+ 287us) client_negotiation.cc:240] Negotiation successful
0809 00:52:46.321125 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":345,"threads_started":1}
I20260809 00:52:46.324101 17591 heartbeater.cc:347] Connected to a master server at 127.9.225.62:39131
I20260809 00:52:46.324486 17591 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:46.325268 17591 heartbeater.cc:511] Master 127.9.225.62:39131 requested a full tablet report, sending...
I20260809 00:52:46.327068 16881 ts_manager.cc:194] Registered new tserver with Master: e0e206a157804986b89c15b0f4ce080a (127.9.225.3:37513)
I20260809 00:52:46.327958 16881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:52199
I20260809 00:52:46.332631 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:52:46.352470 17593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.337820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.9.225.62:39131)
0809 00:52:46.338040 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:46.338045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:46.338637 (+ 592us) server_negotiation.cc:407] Connection header received
0809 00:52:46.338853 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:46.338859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:46.338935 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:46.339039 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:46.340650 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.341539 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.342934 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.343414 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.343529 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.346421 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:46.346428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:46.346431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:46.346474 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:46.349512 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:46.350469 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:46.350481 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:46.350483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:46.350641 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:46.351063 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:46.351067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:46.351070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:46.351454 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:46.351686 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:46.352010 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 00:52:46.352216 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 00:52:46.357429 16881 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:52:46.364841 16881 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:52:46.405992 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.379481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:37513 (local address 127.0.0.1:42228)
0809 00:52:46.395065 (+ 15584us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:46.395086 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:46.395229 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:46.395978 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:46.395986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:46.396023 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:46.396504 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:46.396517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.398994 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.399000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:46.402631 (+ 3631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:46.402645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.404738 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.404745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:46.404990 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.405070 (+ 80us) client_negotiation.cc:769] Sending connection context
0809 00:52:46.405208 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 00:52:46.405486 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15455,"mutex_wait_us":1,"thread_start_us":15647,"threads_started":1}
I20260809 00:52:46.406917 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.379057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:34095 (local address 127.0.0.1:41314)
0809 00:52:46.393589 (+ 14532us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:46.393668 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:46.393935 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:46.395548 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:46.395556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:46.395588 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:46.396151 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:46.396169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.400012 (+ 3843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.400018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:46.402264 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:46.402278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.405968 (+ 3690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.405978 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:46.406200 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.406232 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 00:52:46.406298 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 00:52:46.406446 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14103,"mutex_wait_us":34,"thread_start_us":14063,"threads_started":1}
I20260809 00:52:46.408002 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.379979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42228 (local address 127.9.225.3:37513)
0809 00:52:46.392987 (+ 13008us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:46.393002 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:46.395381 (+ 2379us) server_negotiation.cc:407] Connection header received
0809 00:52:46.395513 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:46.395521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:46.395747 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:46.396716 (+ 969us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:46.397105 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.398802 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.402837 (+ 4035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.404474 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.407005 (+ 2531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.407299 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:46.407410 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 00:52:46.407639 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12688,"thread_start_us":12480,"threads_started":1}
I20260809 00:52:46.408231 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.395577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:41265 (local address 127.0.0.1:60616)
0809 00:52:46.399238 (+ 3661us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:46.399254 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:46.399419 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:46.400358 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:46.400362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:46.400394 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:46.401001 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:46.401013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.402986 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.402991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:46.405225 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:46.405240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.407433 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.407440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:46.407623 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.407656 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 00:52:46.407738 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 00:52:46.407874 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3529,"thread_start_us":3362,"threads_started":1}
I20260809 00:52:46.408332 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.379901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.9.225.1:34095)
0809 00:52:46.394780 (+ 14879us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:46.394791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:46.394830 (+ 39us) server_negotiation.cc:407] Connection header received
0809 00:52:46.394965 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:46.394974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:46.395279 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:46.398016 (+ 2737us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:46.398279 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.399819 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.402448 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.405681 (+ 3233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.407482 (+ 1801us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.407725 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:46.407806 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 00:52:46.408001 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14658,"thread_start_us":14490,"threads_started":1}
I20260809 00:52:46.409109 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.394880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.9.225.2:41265)
0809 00:52:46.398875 (+ 3995us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:46.398887 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:46.399369 (+ 482us) server_negotiation.cc:407] Connection header received
0809 00:52:46.399607 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:46.399618 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:46.399909 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:46.400121 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:46.401196 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.402797 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.405452 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.406916 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.407043 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.407273 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:46.408587 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0809 00:52:46.408768 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3807,"thread_start_us":3649,"threads_started":1}
I20260809 00:52:46.412536 17065 tablet_service.cc:1467] Processing CreateTablet for tablet c224e544421f40bfa145d925d3434b10 (DEFAULT_TABLE table=test-table [id=b9c95904120444508d435c602707618f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:46.412578 17491 tablet_service.cc:1467] Processing CreateTablet for tablet c224e544421f40bfa145d925d3434b10 (DEFAULT_TABLE table=test-table [id=b9c95904120444508d435c602707618f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:46.414273 17065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c224e544421f40bfa145d925d3434b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:46.414296 17491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c224e544421f40bfa145d925d3434b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:46.413730 17288 tablet_service.cc:1467] Processing CreateTablet for tablet c224e544421f40bfa145d925d3434b10 (DEFAULT_TABLE table=test-table [id=b9c95904120444508d435c602707618f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:46.415338 17288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c224e544421f40bfa145d925d3434b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:46.433475 17619 tablet_bootstrap.cc:492] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09: Bootstrap starting.
I20260809 00:52:46.436142 17620 tablet_bootstrap.cc:492] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65: Bootstrap starting.
I20260809 00:52:46.437191 17621 tablet_bootstrap.cc:492] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a: Bootstrap starting.
I20260809 00:52:46.438303 17619 tablet_bootstrap.cc:654] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:46.439888 17619 log.cc:824] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:46.440977 17620 tablet_bootstrap.cc:654] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:46.441258 17621 tablet_bootstrap.cc:654] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:46.442203 17620 log.cc:824] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:46.442608 17621 log.cc:824] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:46.443742 17619 tablet_bootstrap.cc:492] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09: No bootstrap required, opened a new log
I20260809 00:52:46.444118 17619 ts_tablet_manager.cc:1397] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260809 00:52:46.445715 17620 tablet_bootstrap.cc:492] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65: No bootstrap required, opened a new log
I20260809 00:52:46.445868 17621 tablet_bootstrap.cc:492] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a: No bootstrap required, opened a new log
I20260809 00:52:46.446055 17620 ts_tablet_manager.cc:1397] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 00:52:46.446167 17621 ts_tablet_manager.cc:1397] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 00:52:46.454972 17621 raft_consensus.cc:348] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 34095 } } peers { permanent_uuid: "e0e206a157804986b89c15b0f4ce080a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37513 } } peers { permanent_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41265 } }
I20260809 00:52:46.455587 17621 raft_consensus.cc:374] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:46.455708 17621 raft_consensus.cc:729] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e206a157804986b89c15b0f4ce080a, State: Initialized, Role: FOLLOWER
I20260809 00:52:46.456494 17621 consensus_queue.cc:260] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 34095 } } peers { permanent_uuid: "e0e206a157804986b89c15b0f4ce080a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37513 } } peers { permanent_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41265 } }
I20260809 00:52:46.457870 17591 heartbeater.cc:503] Master 127.9.225.62:39131 was elected leader, sending a full tablet report...
I20260809 00:52:46.457934 17620 raft_consensus.cc:348] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 34095 } } peers { permanent_uuid: "e0e206a157804986b89c15b0f4ce080a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37513 } } peers { permanent_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41265 } }
I20260809 00:52:46.457934 17619 raft_consensus.cc:348] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 34095 } } peers { permanent_uuid: "e0e206a157804986b89c15b0f4ce080a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37513 } } peers { permanent_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41265 } }
I20260809 00:52:46.458434 17619 raft_consensus.cc:374] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:46.458434 17620 raft_consensus.cc:374] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:46.458540 17619 raft_consensus.cc:729] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c13ddfa9d1547dfb367b2dd2b01da09, State: Initialized, Role: FOLLOWER
I20260809 00:52:46.458541 17620 raft_consensus.cc:729] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0913aaf5aa245eab06ba7eb4ef63e65, State: Initialized, Role: FOLLOWER
I20260809 00:52:46.458721 17621 ts_tablet_manager.cc:1428] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260809 00:52:46.459244 17620 consensus_queue.cc:260] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 34095 } } peers { permanent_uuid: "e0e206a157804986b89c15b0f4ce080a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37513 } } peers { permanent_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41265 } }
I20260809 00:52:46.459241 17619 consensus_queue.cc:260] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 34095 } } peers { permanent_uuid: "e0e206a157804986b89c15b0f4ce080a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37513 } } peers { permanent_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41265 } }
I20260809 00:52:46.460497 17384 heartbeater.cc:503] Master 127.9.225.62:39131 was elected leader, sending a full tablet report...
I20260809 00:52:46.461594 17620 ts_tablet_manager.cc:1428] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260809 00:52:46.469015 17619 ts_tablet_manager.cc:1428] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09: Time spent starting tablet: real 0.025s user 0.023s sys 0.000s
I20260809 00:52:46.540086 17625 raft_consensus.cc:482] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:52:46.540406 17625 raft_consensus.cc:504] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 34095 } } peers { permanent_uuid: "e0e206a157804986b89c15b0f4ce080a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37513 } } peers { permanent_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41265 } }
I20260809 00:52:46.542284 17625 leader_election.cc:288] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c13ddfa9d1547dfb367b2dd2b01da09 (127.9.225.1:34095), b0913aaf5aa245eab06ba7eb4ef63e65 (127.9.225.2:41265)
W20260809 00:52:46.549052 17592 tablet.cc:2369] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:52:46.549525 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.542809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:33665 (local address 127.9.225.1:34095)
0809 00:52:46.543063 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:46.543073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:46.543098 (+ 25us) server_negotiation.cc:407] Connection header received
0809 00:52:46.543283 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:46.543291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:46.543382 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:46.543490 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:46.544034 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.544957 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.547149 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.548439 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.548631 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.548749 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:46.549028 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 00:52:46.549210 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 00:52:46.549559 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.542738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:34095 (local address 127.9.225.3:33665)
0809 00:52:46.542967 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:46.542986 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:46.543118 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:46.543551 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:46.543556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:46.543591 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:46.543869 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:46.543880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.545137 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.545142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:46.546965 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:46.546979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.548641 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.548647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:46.548843 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.548869 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 00:52:46.549105 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 00:52:46.549253 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 00:52:46.550967 17629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.543206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:41265 (local address 127.9.225.3:49511)
0809 00:52:46.544087 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:46.544106 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:46.544235 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:46.544554 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:46.544558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:46.544584 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:52:46.544945 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:46.544957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.546399 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.546403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:46.548309 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:46.548327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.550213 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.550221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:46.550419 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.550449 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 00:52:46.550544 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 00:52:46.550686 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":542,"threads_started":1}
I20260809 00:52:46.551023 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.543349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:49511 (local address 127.9.225.2:41265)
0809 00:52:46.543558 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:46.543564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:46.544184 (+ 620us) server_negotiation.cc:407] Connection header received
0809 00:52:46.544369 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:46.544374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:46.544441 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:46.544532 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:52:46.545111 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.546225 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.548497 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.549810 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.549970 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.550073 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:46.550598 (+ 525us) server_negotiation.cc:299] Negotiation successful
0809 00:52:46.550742 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 00:52:46.550784 17102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c224e544421f40bfa145d925d3434b10" candidate_uuid: "e0e206a157804986b89c15b0f4ce080a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09" is_pre_election: true
I20260809 00:52:46.551393 17102 raft_consensus.cc:2454] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e206a157804986b89c15b0f4ce080a in term 0.
I20260809 00:52:46.552182 17416 leader_election.cc:302] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c13ddfa9d1547dfb367b2dd2b01da09, e0e206a157804986b89c15b0f4ce080a; no voters:
I20260809 00:52:46.552614 17625 raft_consensus.cc:2781] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:52:46.552759 17625 raft_consensus.cc:482] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:52:46.552883 17625 raft_consensus.cc:3037] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:46.552784 17314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c224e544421f40bfa145d925d3434b10" candidate_uuid: "e0e206a157804986b89c15b0f4ce080a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65" is_pre_election: true
I20260809 00:52:46.553413 17314 raft_consensus.cc:2454] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e206a157804986b89c15b0f4ce080a in term 0.
I20260809 00:52:46.555738 17625 raft_consensus.cc:504] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 34095 } } peers { permanent_uuid: "e0e206a157804986b89c15b0f4ce080a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37513 } } peers { permanent_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41265 } }
I20260809 00:52:46.556542 17625 leader_election.cc:288] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [CANDIDATE]: Term 1 election: Requested vote from peers 6c13ddfa9d1547dfb367b2dd2b01da09 (127.9.225.1:34095), b0913aaf5aa245eab06ba7eb4ef63e65 (127.9.225.2:41265)
I20260809 00:52:46.557149 17102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c224e544421f40bfa145d925d3434b10" candidate_uuid: "e0e206a157804986b89c15b0f4ce080a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09"
I20260809 00:52:46.557163 17314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c224e544421f40bfa145d925d3434b10" candidate_uuid: "e0e206a157804986b89c15b0f4ce080a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65"
I20260809 00:52:46.557418 17102 raft_consensus.cc:3037] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:46.557440 17314 raft_consensus.cc:3037] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:46.559849 17314 raft_consensus.cc:2454] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e206a157804986b89c15b0f4ce080a in term 1.
I20260809 00:52:46.559865 17102 raft_consensus.cc:2454] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e206a157804986b89c15b0f4ce080a in term 1.
I20260809 00:52:46.560408 17416 leader_election.cc:302] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c13ddfa9d1547dfb367b2dd2b01da09, e0e206a157804986b89c15b0f4ce080a; no voters:
I20260809 00:52:46.560762 17625 raft_consensus.cc:2781] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:46.561298 17625 raft_consensus.cc:686] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 1 LEADER]: Becoming Leader. State: Replica: e0e206a157804986b89c15b0f4ce080a, State: Running, Role: LEADER
I20260809 00:52:46.561919 17625 consensus_queue.cc:237] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 34095 } } peers { permanent_uuid: "e0e206a157804986b89c15b0f4ce080a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37513 } } peers { permanent_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41265 } }
I20260809 00:52:46.567054 16880 catalog_manager.cc:5495] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e206a157804986b89c15b0f4ce080a (127.9.225.3). New cstate: current_term: 1 leader_uuid: "e0e206a157804986b89c15b0f4ce080a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0e206a157804986b89c15b0f4ce080a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41265 } health_report { overall_health: UNKNOWN } } }
I20260809 00:52:46.589890 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.583789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.9.225.3:37513)
0809 00:52:46.584066 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:46.584075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:46.584156 (+ 81us) server_negotiation.cc:407] Connection header received
0809 00:52:46.584433 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:46.584438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:46.584547 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:46.584696 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:52:46.585131 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.586006 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.587411 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.587834 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.587986 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.588548 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:52:46.589181 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:52:46.589396 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:46.589494 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 00:52:46.589594 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260809 00:52:46.599509 17102 raft_consensus.cc:1261] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09 [term 1 FOLLOWER]: Refusing update from remote peer e0e206a157804986b89c15b0f4ce080a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:46.599504 17314 raft_consensus.cc:1261] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65 [term 1 FOLLOWER]: Refusing update from remote peer e0e206a157804986b89c15b0f4ce080a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:46.600703 17625 consensus_queue.cc:1035] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c13ddfa9d1547dfb367b2dd2b01da09" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:46.601138 17630 consensus_queue.cc:1035] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0913aaf5aa245eab06ba7eb4ef63e65" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 00:52:46.611104 17176 tablet.cc:2369] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:52:46.637274 16879 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:46.637667 16879 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:46.642081 16879 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f89afe7936f499998430eec4bb0e959: Sending DeleteTablet for 3 replicas of tablet c224e544421f40bfa145d925d3434b10
I20260809 00:52:46.643592 17491 tablet_service.cc:1514] Processing DeleteTablet for tablet c224e544421f40bfa145d925d3434b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:46 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:42228
I20260809 00:52:46.643688 17288 tablet_service.cc:1514] Processing DeleteTablet for tablet c224e544421f40bfa145d925d3434b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:46 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:60616
I20260809 00:52:46.643911 17065 tablet_service.cc:1514] Processing DeleteTablet for tablet c224e544421f40bfa145d925d3434b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:46 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:41314
I20260809 00:52:46.651258 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 16986
I20260809 00:52:46.656844 17646 tablet_replica.cc:330] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65: stopping tablet replica
I20260809 00:52:46.657644 17646 raft_consensus.cc:2229] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:46.658386 17646 raft_consensus.cc:2258] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:46.661103 17646 ts_tablet_manager.cc:1905] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:46.661857 17647 tablet_replica.cc:330] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a: stopping tablet replica
I20260809 00:52:46.662367 17647 raft_consensus.cc:2229] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:52:46.663213 17647 raft_consensus.cc:2258] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:46.666949 17648 tablet_replica.cc:330] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09: stopping tablet replica
I20260809 00:52:46.667601 17648 raft_consensus.cc:2229] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:46.668138 17647 ts_tablet_manager.cc:1905] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:46.668336 17648 raft_consensus.cc:2258] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:46.670812 17648 ts_tablet_manager.cc:1905] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:46.674211 17646 ts_tablet_manager.cc:1918] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:52:46.674453 17646 log.cc:1196] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/c224e544421f40bfa145d925d3434b10
I20260809 00:52:46.675094 17646 ts_tablet_manager.cc:1939] T c224e544421f40bfa145d925d3434b10 P b0913aaf5aa245eab06ba7eb4ef63e65: Deleting consensus metadata
I20260809 00:52:46.676340 17647 ts_tablet_manager.cc:1918] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:52:46.676560 17647 log.cc:1196] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/c224e544421f40bfa145d925d3434b10
I20260809 00:52:46.677321 17647 ts_tablet_manager.cc:1939] T c224e544421f40bfa145d925d3434b10 P e0e206a157804986b89c15b0f4ce080a: Deleting consensus metadata
I20260809 00:52:46.677495 16864 catalog_manager.cc:4841] TS b0913aaf5aa245eab06ba7eb4ef63e65 (127.9.225.2:41265): tablet c224e544421f40bfa145d925d3434b10 (table test-table [id=b9c95904120444508d435c602707618f]) successfully deleted
I20260809 00:52:46.679486 16863 catalog_manager.cc:4841] TS e0e206a157804986b89c15b0f4ce080a (127.9.225.3:37513): tablet c224e544421f40bfa145d925d3434b10 (table test-table [id=b9c95904120444508d435c602707618f]) successfully deleted
I20260809 00:52:46.679560 17648 ts_tablet_manager.cc:1918] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:52:46.679922 17648 log.cc:1196] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/c224e544421f40bfa145d925d3434b10
I20260809 00:52:46.680501 17648 ts_tablet_manager.cc:1939] T c224e544421f40bfa145d925d3434b10 P 6c13ddfa9d1547dfb367b2dd2b01da09: Deleting consensus metadata
I20260809 00:52:46.682384 16864 catalog_manager.cc:4841] TS 6c13ddfa9d1547dfb367b2dd2b01da09 (127.9.225.1:34095): tablet c224e544421f40bfa145d925d3434b10 (table test-table [id=b9c95904120444508d435c602707618f]) successfully deleted
Aug 09 00:52:46 dist-test-slave-j2vl krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:46.694607 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.652953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.9.225.1:34095)
0809 00:52:46.653201 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:46.653208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:46.666148 (+ 12940us) server_negotiation.cc:407] Connection header received
0809 00:52:46.666375 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:46.666381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:46.666491 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:46.666633 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:46.677999 (+ 11366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.679238 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.681752 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.682457 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.682607 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.687949 (+ 5342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:46.687969 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:46.687988 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:46.688034 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:46.691238 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:46.692184 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:46.692194 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:46.692197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:46.692371 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:46.692820 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:46.692826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:46.692829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:46.693247 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:46.693658 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:46.693951 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 00:52:46.694206 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 00:52:46.695379 17165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:46.794687 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 16986
I20260809 00:52:46.830641 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 17184
Aug 09 00:52:46 dist-test-slave-j2vl krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:52:46.854239 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.831572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.9.225.2:41265)
0809 00:52:46.831776 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:46.831785 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:46.831812 (+ 27us) server_negotiation.cc:407] Connection header received
0809 00:52:46.832067 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:46.832077 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:46.832169 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:46.832298 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:46.834130 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.835098 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.836607 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.837106 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.837330 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.844100 (+ 6770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:46.844133 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:46.844167 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:46.844241 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:46.849631 (+ 5390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:46.850909 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:46.850924 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:46.850928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:46.851144 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:46.851636 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:46.851641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:46.851645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:46.852290 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:46.852662 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:46.853501 (+ 839us) server_negotiation.cc:299] Negotiation successful
0809 00:52:46.853824 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 00:52:46.855144 17372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:46.950500 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 17184
I20260809 00:52:46.986835 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 17393
Aug 09 00:52:46 dist-test-slave-j2vl krb5kdc[16819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:52:47.005658 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:46.987776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.9.225.3:37513)
0809 00:52:46.987961 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:46.987969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:46.988013 (+ 44us) server_negotiation.cc:407] Connection header received
0809 00:52:46.988241 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:46.988251 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:46.988327 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:46.988463 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:46.989918 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.990800 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.992115 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:46.992613 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:46.992776 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:46.997737 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:46.997765 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:46.997788 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:46.997859 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:47.002243 (+ 4384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:47.003147 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:47.003157 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:47.003159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:47.003329 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:47.003788 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:47.003798 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:47.003801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:47.004237 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:47.004635 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:47.004949 (+ 314us) server_negotiation.cc:299] Negotiation successful
0809 00:52:47.005209 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 00:52:47.006436 17583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:47.140105 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 17393
I20260809 00:52:47.183359 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 16837
I20260809 00:52:47.184623 16928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:47.299086 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 16837
2026-08-09T00:52:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5072 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T00:52:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:47Z Disabled control of system clock
I20260809 00:52:47.391423 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:33049
--webserver_interface=127.9.225.62
--webserver_port=0
--builtin_ntp_servers=127.9.225.20:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:33049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 00:52:47.777558 17661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:47.777963 17661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:47.778141 17661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:47.786458 17661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:47.786623 17661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:47.786664 17661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:47.786718 17661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:47.786763 17661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:52:47.796895 17661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:33049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:33049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:47.799129 17661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:47.801229 17661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:47.816502 17668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:47.816561 17672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:47.816596 17669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:47.817915 17661 server_base.cc:1034] running on GCE node
I20260809 00:52:47.818945 17661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:47.820806 17661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:47.822080 17661 hybrid_clock.cc:648] HybridClock initialized: now 1786236767822036 us; error 41 us; skew 500 ppm
I20260809 00:52:47.822494 17661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:52:47.825213 17661 webserver.cc:460] Webserver started at http://127.9.225.62:36865/ using document root <none> and password file <none>
I20260809 00:52:47.826086 17661 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:47.826241 17661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:47.826635 17661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:47.830148 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "d555e466bf394e11a094923e45ca0299"
format_stamp: "Formatted at 2026-08-09 00:52:47 on dist-test-slave-j2vl"
server_key: "94b6b275724cb04bc82f847488119bde"
server_key_iv: "122a7d4d758e4a9e6e3cb63f78edf7ca"
server_key_version: "encryptionkey@0"
I20260809 00:52:47.831146 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "d555e466bf394e11a094923e45ca0299"
format_stamp: "Formatted at 2026-08-09 00:52:47 on dist-test-slave-j2vl"
server_key: "94b6b275724cb04bc82f847488119bde"
server_key_iv: "122a7d4d758e4a9e6e3cb63f78edf7ca"
server_key_version: "encryptionkey@0"
I20260809 00:52:47.837954 17661 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260809 00:52:47.842696 17682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:47.844431 17661 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:52:47.844682 17661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "d555e466bf394e11a094923e45ca0299"
format_stamp: "Formatted at 2026-08-09 00:52:47 on dist-test-slave-j2vl"
server_key: "94b6b275724cb04bc82f847488119bde"
server_key_iv: "122a7d4d758e4a9e6e3cb63f78edf7ca"
server_key_version: "encryptionkey@0"
I20260809 00:52:47.844973 17661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:47.852252 17661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:47.853256 17661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:47.853647 17661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:47.875391 17661 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:33049
I20260809 00:52:47.875414 17766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:33049 every 8 connection(s)
I20260809 00:52:47.877025 17661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:47.880957 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 17661
I20260809 00:52:47.881179 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:47.881879 17767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:47.882121 10116 external_mini_cluster.cc:1428] Setting key be9c985f58669a61e205ae5ea23bb1f4
I20260809 00:52:47.892474 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:47.884496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.9.225.62:33049)
0809 00:52:47.885806 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:47.885821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:47.885871 (+ 50us) server_negotiation.cc:407] Connection header received
0809 00:52:47.887567 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:47.887599 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:47.888161 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:47.888669 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:47.890091 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:47.890100 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:47.890126 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:47.890150 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:47.890246 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:47.890276 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:47.890562 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:47.890727 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 00:52:47.891106 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":512,"threads_started":1}
I20260809 00:52:47.902277 17767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299: Bootstrap starting.
I20260809 00:52:47.906103 17767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:47.907298 17767 log.cc:824] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:47.910931 17767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299: No bootstrap required, opened a new log
I20260809 00:52:47.919726 17767 raft_consensus.cc:348] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d555e466bf394e11a094923e45ca0299" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33049 } }
I20260809 00:52:47.920233 17767 raft_consensus.cc:374] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:47.920358 17767 raft_consensus.cc:729] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d555e466bf394e11a094923e45ca0299, State: Initialized, Role: FOLLOWER
I20260809 00:52:47.921030 17767 consensus_queue.cc:260] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d555e466bf394e11a094923e45ca0299" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33049 } }
I20260809 00:52:47.921315 17767 raft_consensus.cc:388] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:47.921533 17767 raft_consensus.cc:482] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:47.921727 17767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:47.924314 17767 raft_consensus.cc:504] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d555e466bf394e11a094923e45ca0299" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33049 } }
I20260809 00:52:47.925006 17767 leader_election.cc:302] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d555e466bf394e11a094923e45ca0299; no voters:
I20260809 00:52:47.925855 17767 leader_election.cc:288] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:47.926043 17774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:47.926844 17774 raft_consensus.cc:686] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [term 1 LEADER]: Becoming Leader. State: Replica: d555e466bf394e11a094923e45ca0299, State: Running, Role: LEADER
I20260809 00:52:47.927472 17774 consensus_queue.cc:237] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d555e466bf394e11a094923e45ca0299" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33049 } }
I20260809 00:52:47.928613 17767 sys_catalog.cc:564] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:47.930830 17776 sys_catalog.cc:455] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d555e466bf394e11a094923e45ca0299. Latest consensus state: current_term: 1 leader_uuid: "d555e466bf394e11a094923e45ca0299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d555e466bf394e11a094923e45ca0299" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33049 } } }
I20260809 00:52:47.931560 17776 sys_catalog.cc:458] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:47.930824 17775 sys_catalog.cc:455] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d555e466bf394e11a094923e45ca0299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d555e466bf394e11a094923e45ca0299" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33049 } } }
I20260809 00:52:47.931933 17775 sys_catalog.cc:458] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:47.935362 17784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:47.941659 17784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:47.952313 17784 catalog_manager.cc:1347] Generated new cluster ID: 26da2907975e495d89c5dabf36d7308b
I20260809 00:52:47.952478 17784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:47.966239 17784 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:47.966823 17784 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:47.984982 17784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299: Generated new TSK 0
I20260809 00:52:47.986285 17784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:52:48.000594 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33049
--builtin_ntp_servers=127.9.225.20:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 00:52:48.387023 17802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:48.387470 17802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:48.387619 17802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:48.395949 17802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:48.396101 17802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:48.396291 17802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:52:48.406522 17802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:48.408293 17802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:48.410516 17802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:48.425838 17813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:48.425909 17816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:48.425885 17814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:48.426798 17802 server_base.cc:1034] running on GCE node
I20260809 00:52:48.427511 17802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:48.428658 17802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:48.429920 17802 hybrid_clock.cc:648] HybridClock initialized: now 1786236768429830 us; error 66 us; skew 500 ppm
I20260809 00:52:48.430294 17802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:52:48.432958 17802 webserver.cc:460] Webserver started at http://127.9.225.1:35961/ using document root <none> and password file <none>
I20260809 00:52:48.433848 17802 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:48.434000 17802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:48.434362 17802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:48.437603 17802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "e4e2a259a353437599c2cd007b56e596"
format_stamp: "Formatted at 2026-08-09 00:52:48 on dist-test-slave-j2vl"
server_key: "5ec44286bf5bae62d3265d6a99a8fd20"
server_key_iv: "5829716481589504e948940d7c5292f6"
server_key_version: "encryptionkey@0"
I20260809 00:52:48.438521 17802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "e4e2a259a353437599c2cd007b56e596"
format_stamp: "Formatted at 2026-08-09 00:52:48 on dist-test-slave-j2vl"
server_key: "5ec44286bf5bae62d3265d6a99a8fd20"
server_key_iv: "5829716481589504e948940d7c5292f6"
server_key_version: "encryptionkey@0"
I20260809 00:52:48.444485 17802 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260809 00:52:48.448669 17825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:48.450562 17802 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260809 00:52:48.450824 17802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "e4e2a259a353437599c2cd007b56e596"
format_stamp: "Formatted at 2026-08-09 00:52:48 on dist-test-slave-j2vl"
server_key: "5ec44286bf5bae62d3265d6a99a8fd20"
server_key_iv: "5829716481589504e948940d7c5292f6"
server_key_version: "encryptionkey@0"
I20260809 00:52:48.451105 17802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:48.457540 17802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:48.458501 17802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:48.458809 17802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:48.459895 17802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:48.461694 17802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:48.461809 17802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:48.461941 17802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:48.462023 17802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:48.501132 17802 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:41021
I20260809 00:52:48.501175 17990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:41021 every 8 connection(s)
I20260809 00:52:48.502591 17802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:52:48.509964 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 17802
I20260809 00:52:48.510195 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:52:48.510869 10116 external_mini_cluster.cc:1428] Setting key 74ee68ac95718448f90c7740b382d70a
I20260809 00:52:48.511615 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:48.505412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:46765 (local address 127.9.225.62:33049)
0809 00:52:48.505898 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:48.505903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:48.507195 (+ 1292us) server_negotiation.cc:407] Connection header received
0809 00:52:48.508995 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:48.509001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:48.509084 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:48.509207 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:48.510745 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:48.510751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:48.510756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:48.510782 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:48.510826 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:48.510840 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:48.510982 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:48.511195 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 00:52:48.511274 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":239,"threads_started":1}
I20260809 00:52:48.512887 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:48.506050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33049 (local address 127.9.225.1:46765)
0809 00:52:48.507023 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:48.507062 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:48.508698 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:48.509515 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:48.509524 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:48.509623 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:48.510419 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:52:48.510422 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:48.510463 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:52:48.510558 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:48.510958 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:48.510970 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:48.510978 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 00:52:48.511296 (+ 318us) client_negotiation.cc:240] Negotiation successful
0809 00:52:48.511589 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":310,"threads_started":1}
I20260809 00:52:48.514076 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33049
--builtin_ntp_servers=127.9.225.20:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 00:52:48.514706 17991 heartbeater.cc:347] Connected to a master server at 127.9.225.62:33049
I20260809 00:52:48.515060 17991 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:48.515695 17991 heartbeater.cc:511] Master 127.9.225.62:33049 requested a full tablet report, sending...
I20260809 00:52:48.517982 17707 ts_manager.cc:194] Registered new tserver with Master: e4e2a259a353437599c2cd007b56e596 (127.9.225.1:41021)
W20260809 00:52:48.906071 17996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:48.906412 17996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:48.906580 17996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:48.914511 17996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:48.914644 17996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:48.914844 17996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:52:48.924814 17996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:48.926422 17996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:48.928494 17996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:48.943162 18001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:48.943198 18002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:48.943866 18004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:48.944489 17996 server_base.cc:1034] running on GCE node
I20260809 00:52:48.945143 17996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:48.946321 17996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:48.947547 17996 hybrid_clock.cc:648] HybridClock initialized: now 1786236768947498 us; error 44 us; skew 500 ppm
I20260809 00:52:48.948002 17996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:52:48.950822 17996 webserver.cc:460] Webserver started at http://127.9.225.2:41611/ using document root <none> and password file <none>
I20260809 00:52:48.951718 17996 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:48.951822 17996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:48.952153 17996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:48.955608 17996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "c2715406f80c4f35ab51dd9575106e2b"
format_stamp: "Formatted at 2026-08-09 00:52:48 on dist-test-slave-j2vl"
server_key: "053f9e5db74dee77b04e56b53c9fdf78"
server_key_iv: "ee8629d8df553d9012bc84139f3b9f63"
server_key_version: "encryptionkey@0"
I20260809 00:52:48.956596 17996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "c2715406f80c4f35ab51dd9575106e2b"
format_stamp: "Formatted at 2026-08-09 00:52:48 on dist-test-slave-j2vl"
server_key: "053f9e5db74dee77b04e56b53c9fdf78"
server_key_iv: "ee8629d8df553d9012bc84139f3b9f63"
server_key_version: "encryptionkey@0"
I20260809 00:52:48.962921 17996 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 00:52:48.967129 18013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:48.968865 17996 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:52:48.969053 17996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "c2715406f80c4f35ab51dd9575106e2b"
format_stamp: "Formatted at 2026-08-09 00:52:48 on dist-test-slave-j2vl"
server_key: "053f9e5db74dee77b04e56b53c9fdf78"
server_key_iv: "ee8629d8df553d9012bc84139f3b9f63"
server_key_version: "encryptionkey@0"
I20260809 00:52:48.969275 17996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:48.977193 17996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:48.978207 17996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:48.978472 17996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:48.979569 17996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:48.981199 17996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:48.981294 17996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:48.981457 17996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:48.981523 17996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:49.018627 17996 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:39241
I20260809 00:52:49.018692 18182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:39241 every 8 connection(s)
I20260809 00:52:49.020253 17996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:52:49.023411 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 17996
I20260809 00:52:49.023618 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:52:49.024147 10116 external_mini_cluster.cc:1428] Setting key 2f15b4779d67c45d9a647c9f16b5f552
I20260809 00:52:49.027196 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33049
--builtin_ntp_servers=127.9.225.20:36215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 00:52:49.030550 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.023283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:39977 (local address 127.9.225.62:33049)
0809 00:52:49.023853 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:49.023860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:49.025467 (+ 1607us) server_negotiation.cc:407] Connection header received
0809 00:52:49.027547 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:49.027560 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:49.027624 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:49.027767 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:49.029502 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:49.029509 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:49.029515 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:49.029548 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:49.029600 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:49.029615 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:49.029785 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:49.030034 (+ 249us) server_negotiation.cc:299] Negotiation successful
0809 00:52:49.030124 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":273,"threads_started":1}
I20260809 00:52:49.031699 18186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.023976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33049 (local address 127.9.225.2:39977)
0809 00:52:49.025252 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:49.025321 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:49.027108 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:49.028085 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:49.028095 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:49.028213 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:49.029036 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:52:49.029039 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:49.029127 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:52:49.029259 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:49.029737 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:49.029752 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:49.029764 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 00:52:49.030125 (+ 361us) client_negotiation.cc:240] Negotiation successful
0809 00:52:49.030463 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":418,"threads_started":1}
I20260809 00:52:49.033896 18183 heartbeater.cc:347] Connected to a master server at 127.9.225.62:33049
I20260809 00:52:49.034328 18183 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:49.035084 18183 heartbeater.cc:511] Master 127.9.225.62:33049 requested a full tablet report, sending...
I20260809 00:52:49.036916 17707 ts_manager.cc:194] Registered new tserver with Master: c2715406f80c4f35ab51dd9575106e2b (127.9.225.2:39241)
W20260809 00:52:49.405136 18188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:49.405575 18188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:49.405682 18188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:49.413519 18188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:49.413642 18188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:49.413761 18188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:52:49.423949 18188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:49.425478 18188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:49.427510 18188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:49.442942 18196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:49.442963 18198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:49.442942 18195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:49.443822 18188 server_base.cc:1034] running on GCE node
I20260809 00:52:49.444587 18188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:49.445719 18188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:49.447023 18188 hybrid_clock.cc:648] HybridClock initialized: now 1786236769446947 us; error 66 us; skew 500 ppm
I20260809 00:52:49.447465 18188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:52:49.451004 18188 webserver.cc:460] Webserver started at http://127.9.225.3:37747/ using document root <none> and password file <none>
I20260809 00:52:49.452045 18188 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:49.452199 18188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:49.452603 18188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:49.456493 18188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "5f056cb36721414ca7eaaf6e301d9500"
format_stamp: "Formatted at 2026-08-09 00:52:49 on dist-test-slave-j2vl"
server_key: "35ddc3e43e288d3e70d071aba7528493"
server_key_iv: "5f7435cb7dabeab7d4358180a10b1c70"
server_key_version: "encryptionkey@0"
I20260809 00:52:49.457554 18188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "5f056cb36721414ca7eaaf6e301d9500"
format_stamp: "Formatted at 2026-08-09 00:52:49 on dist-test-slave-j2vl"
server_key: "35ddc3e43e288d3e70d071aba7528493"
server_key_iv: "5f7435cb7dabeab7d4358180a10b1c70"
server_key_version: "encryptionkey@0"
I20260809 00:52:49.464250 18188 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:52:49.469521 18209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:49.471573 18188 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260809 00:52:49.471848 18188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "5f056cb36721414ca7eaaf6e301d9500"
format_stamp: "Formatted at 2026-08-09 00:52:49 on dist-test-slave-j2vl"
server_key: "35ddc3e43e288d3e70d071aba7528493"
server_key_iv: "5f7435cb7dabeab7d4358180a10b1c70"
server_key_version: "encryptionkey@0"
I20260809 00:52:49.472164 18188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:49.478984 18188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:49.480013 18188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:49.480333 18188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:49.481498 18188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:49.483107 18188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:49.483247 18188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:49.483384 18188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:49.483481 18188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:49.519666 18188 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:46717
I20260809 00:52:49.519666 18381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:46717 every 8 connection(s)
I20260809 00:52:49.521514 18188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:52:49.521646 17991 heartbeater.cc:503] Master 127.9.225.62:33049 was elected leader, sending a full tablet report...
I20260809 00:52:49.527844 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 18188
I20260809 00:52:49.528049 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:52:49.528533 10116 external_mini_cluster.cc:1428] Setting key 1ff7e9ce1402a7145afa5b818d78aeb9
I20260809 00:52:49.531806 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.524855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:42243 (local address 127.9.225.62:33049)
0809 00:52:49.525046 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:49.525051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:49.526818 (+ 1767us) server_negotiation.cc:407] Connection header received
0809 00:52:49.528983 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:49.528990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:49.529099 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:49.529252 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:49.531002 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:49.531008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:49.531013 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:49.531037 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:49.531081 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:49.531094 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:49.531224 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:49.531454 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 00:52:49.531539 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 00:52:49.533004 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.525568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33049 (local address 127.9.225.3:42243)
0809 00:52:49.526636 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:49.526686 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:49.528545 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:49.529580 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:49.529615 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:49.529773 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:49.530633 (+ 860us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:52:49.530636 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:49.530686 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:52:49.530797 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:49.531198 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:49.531211 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:49.531221 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 00:52:49.531590 (+ 369us) client_negotiation.cc:240] Negotiation successful
0809 00:52:49.531913 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":338,"threads_started":1}
I20260809 00:52:49.534814 18382 heartbeater.cc:347] Connected to a master server at 127.9.225.62:33049
I20260809 00:52:49.535192 18382 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:49.535967 18382 heartbeater.cc:511] Master 127.9.225.62:33049 requested a full tablet report, sending...
I20260809 00:52:49.537792 17707 ts_manager.cc:194] Registered new tserver with Master: 5f056cb36721414ca7eaaf6e301d9500 (127.9.225.3:46717)
I20260809 00:52:49.544939 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:52:49.552791 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.550027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.9.225.62:33049)
0809 00:52:49.550209 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:49.550214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:49.550770 (+ 556us) server_negotiation.cc:407] Connection header received
0809 00:52:49.550924 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:49.550929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:49.550991 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:49.551086 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:49.552220 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:49.552225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:49.552228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:49.552246 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:49.552279 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:49.552292 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:49.552404 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:49.552484 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 00:52:49.552538 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 00:52:49.557577 17707 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:52:49.564956 17707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:52:49.599386 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.580165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:39241 (local address 127.0.0.1:54142)
0809 00:52:49.581743 (+ 1578us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:49.581806 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:49.582000 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:49.596333 (+ 14333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:49.596344 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:49.596398 (+ 54us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:49.597971 (+ 1573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:52:49.597977 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:49.598001 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:52:49.598054 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:49.598618 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:49.598626 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:49.598633 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 00:52:49.598787 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 00:52:49.598874 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"thread_start_us":16083,"threads_started":1}
I20260809 00:52:49.599448 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.583374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.9.225.3:46717)
0809 00:52:49.595599 (+ 12225us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:49.595609 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:49.595650 (+ 41us) server_negotiation.cc:407] Connection header received
0809 00:52:49.595732 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:49.595741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:49.595991 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:49.596192 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:49.597931 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:49.597942 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:49.597961 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:49.597985 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:49.598055 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:49.598075 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:49.598319 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:49.598931 (+ 612us) server_negotiation.cc:299] Negotiation successful
0809 00:52:49.599054 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12041,"thread_start_us":11818,"threads_started":1}
I20260809 00:52:49.599495 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.582922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:46717 (local address 127.0.0.1:33782)
0809 00:52:49.594779 (+ 11857us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:49.594805 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:49.594927 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:49.596283 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:49.596293 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:49.596399 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:49.597512 (+ 1113us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:52:49.597516 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:49.597550 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:52:49.597647 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:49.598480 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:49.598495 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:49.598503 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 00:52:49.598729 (+ 226us) client_negotiation.cc:240] Negotiation successful
0809 00:52:49.598939 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11709,"mutex_wait_us":54,"thread_start_us":10728,"threads_started":1}
I20260809 00:52:49.600158 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.580421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.9.225.2:39241)
0809 00:52:49.594487 (+ 14066us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:49.594499 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:49.594540 (+ 41us) server_negotiation.cc:407] Connection header received
0809 00:52:49.594629 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:49.594638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:49.594980 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:49.595167 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:49.598313 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:49.598321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:49.598332 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:49.598391 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:49.598463 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:49.598491 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:49.598653 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:49.599717 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0809 00:52:49.599831 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13866,"thread_start_us":13718,"threads_started":1}
I20260809 00:52:49.601401 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.580679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:41021 (local address 127.0.0.1:56750)
0809 00:52:49.594130 (+ 13451us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:49.594170 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:49.594397 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:49.596797 (+ 2400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:49.596803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:49.596838 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:49.598545 (+ 1707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:52:49.598549 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:49.598569 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:52:49.598612 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:49.599200 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:49.599211 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:49.599232 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 00:52:49.599401 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 00:52:49.599462 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13209,"thread_start_us":1144,"threads_started":1}
I20260809 00:52:49.601894 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.581549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.9.225.1:41021)
0809 00:52:49.590386 (+ 8837us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:49.590417 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:49.594350 (+ 3933us) server_negotiation.cc:407] Connection header received
0809 00:52:49.594582 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:49.594597 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:49.594974 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:49.595184 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:49.598804 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:49.598811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:49.598822 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:49.598840 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:49.598929 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:49.598950 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:49.599137 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:49.600044 (+ 907us) server_negotiation.cc:299] Negotiation successful
0809 00:52:49.600174 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8533,"thread_start_us":8312,"threads_started":1}
I20260809 00:52:49.604483 18086 tablet_service.cc:1467] Processing CreateTablet for tablet 81c7c4d15b764610a6f1485eef9fecab (DEFAULT_TABLE table=test-table [id=d761835f22004be9b0011c0b5953f53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:49.604483 18280 tablet_service.cc:1467] Processing CreateTablet for tablet 81c7c4d15b764610a6f1485eef9fecab (DEFAULT_TABLE table=test-table [id=d761835f22004be9b0011c0b5953f53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:49.606199 18086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c7c4d15b764610a6f1485eef9fecab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:49.606196 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c7c4d15b764610a6f1485eef9fecab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:49.605996 17894 tablet_service.cc:1467] Processing CreateTablet for tablet 81c7c4d15b764610a6f1485eef9fecab (DEFAULT_TABLE table=test-table [id=d761835f22004be9b0011c0b5953f53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:52:49.607309 17894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81c7c4d15b764610a6f1485eef9fecab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:49.619977 18415 tablet_bootstrap.cc:492] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596: Bootstrap starting.
I20260809 00:52:49.622237 18416 tablet_bootstrap.cc:492] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500: Bootstrap starting.
I20260809 00:52:49.626611 18415 tablet_bootstrap.cc:654] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:49.626621 18416 tablet_bootstrap.cc:654] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:49.627712 18417 tablet_bootstrap.cc:492] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b: Bootstrap starting.
I20260809 00:52:49.628132 18416 log.cc:824] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:49.628126 18415 log.cc:824] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:49.632087 18417 tablet_bootstrap.cc:654] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:49.632555 18416 tablet_bootstrap.cc:492] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500: No bootstrap required, opened a new log
I20260809 00:52:49.632886 18416 ts_tablet_manager.cc:1397] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260809 00:52:49.632929 18415 tablet_bootstrap.cc:492] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596: No bootstrap required, opened a new log
I20260809 00:52:49.633203 18415 ts_tablet_manager.cc:1397] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20260809 00:52:49.633594 18417 log.cc:824] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:49.637435 18417 tablet_bootstrap.cc:492] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b: No bootstrap required, opened a new log
I20260809 00:52:49.637760 18417 ts_tablet_manager.cc:1397] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260809 00:52:49.642273 18416 raft_consensus.cc:348] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2715406f80c4f35ab51dd9575106e2b" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39241 } } peers { permanent_uuid: "e4e2a259a353437599c2cd007b56e596" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 41021 } } peers { permanent_uuid: "5f056cb36721414ca7eaaf6e301d9500" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46717 } }
I20260809 00:52:49.642786 18416 raft_consensus.cc:374] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:49.642892 18416 raft_consensus.cc:729] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f056cb36721414ca7eaaf6e301d9500, State: Initialized, Role: FOLLOWER
I20260809 00:52:49.643680 18416 consensus_queue.cc:260] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2715406f80c4f35ab51dd9575106e2b" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39241 } } peers { permanent_uuid: "e4e2a259a353437599c2cd007b56e596" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 41021 } } peers { permanent_uuid: "5f056cb36721414ca7eaaf6e301d9500" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46717 } }
I20260809 00:52:49.644430 18415 raft_consensus.cc:348] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2715406f80c4f35ab51dd9575106e2b" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39241 } } peers { permanent_uuid: "e4e2a259a353437599c2cd007b56e596" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 41021 } } peers { permanent_uuid: "5f056cb36721414ca7eaaf6e301d9500" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46717 } }
I20260809 00:52:49.644929 18415 raft_consensus.cc:374] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:49.644958 18382 heartbeater.cc:503] Master 127.9.225.62:33049 was elected leader, sending a full tablet report...
I20260809 00:52:49.645040 18415 raft_consensus.cc:729] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e2a259a353437599c2cd007b56e596, State: Initialized, Role: FOLLOWER
I20260809 00:52:49.645749 18415 consensus_queue.cc:260] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2715406f80c4f35ab51dd9575106e2b" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39241 } } peers { permanent_uuid: "e4e2a259a353437599c2cd007b56e596" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 41021 } } peers { permanent_uuid: "5f056cb36721414ca7eaaf6e301d9500" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46717 } }
I20260809 00:52:49.647408 18416 ts_tablet_manager.cc:1428] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260809 00:52:49.648425 18415 ts_tablet_manager.cc:1428] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260809 00:52:49.651759 18417 raft_consensus.cc:348] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2715406f80c4f35ab51dd9575106e2b" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39241 } } peers { permanent_uuid: "e4e2a259a353437599c2cd007b56e596" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 41021 } } peers { permanent_uuid: "5f056cb36721414ca7eaaf6e301d9500" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46717 } }
I20260809 00:52:49.652457 18417 raft_consensus.cc:374] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:49.652599 18417 raft_consensus.cc:729] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2715406f80c4f35ab51dd9575106e2b, State: Initialized, Role: FOLLOWER
I20260809 00:52:49.653461 18417 consensus_queue.cc:260] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2715406f80c4f35ab51dd9575106e2b" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39241 } } peers { permanent_uuid: "e4e2a259a353437599c2cd007b56e596" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 41021 } } peers { permanent_uuid: "5f056cb36721414ca7eaaf6e301d9500" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46717 } }
I20260809 00:52:49.654745 18183 heartbeater.cc:503] Master 127.9.225.62:33049 was elected leader, sending a full tablet report...
I20260809 00:52:49.655339 18417 ts_tablet_manager.cc:1428] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260809 00:52:49.755184 17992 tablet.cc:2369] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:52:49.772835 18184 tablet.cc:2369] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:52:49.773826 18383 tablet.cc:2369] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:52:49.773826 18425 raft_consensus.cc:482] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:52:49.774214 18425 raft_consensus.cc:504] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2715406f80c4f35ab51dd9575106e2b" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39241 } } peers { permanent_uuid: "e4e2a259a353437599c2cd007b56e596" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 41021 } } peers { permanent_uuid: "5f056cb36721414ca7eaaf6e301d9500" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46717 } }
I20260809 00:52:49.776417 18425 leader_election.cc:288] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2715406f80c4f35ab51dd9575106e2b (127.9.225.2:39241), 5f056cb36721414ca7eaaf6e301d9500 (127.9.225.3:46717)
I20260809 00:52:49.780584 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.776774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:42567 (local address 127.9.225.2:39241)
0809 00:52:49.777035 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:49.777043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:49.778076 (+ 1033us) server_negotiation.cc:407] Connection header received
0809 00:52:49.778231 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:49.778237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:49.778315 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:49.778431 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:49.779698 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:49.779704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:49.779709 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:49.779732 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:49.779776 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:49.779797 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:49.779937 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:49.780150 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 00:52:49.780229 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260809 00:52:49.780902 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.777014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:48479 (local address 127.9.225.3:46717)
0809 00:52:49.777263 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:49.777285 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:49.778246 (+ 961us) server_negotiation.cc:407] Connection header received
0809 00:52:49.778465 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:49.778472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:49.778543 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:49.778684 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:49.779961 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:49.779967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:49.779972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:49.779997 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:49.780041 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:49.780060 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:49.780197 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:49.780454 (+ 257us) server_negotiation.cc:299] Negotiation successful
0809 00:52:49.780628 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260809 00:52:49.781184 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.776968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:46717 (local address 127.9.225.1:48479)
0809 00:52:49.778073 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:49.778102 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:49.778246 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:49.778731 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:49.778735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:49.778754 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:49.779667 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:52:49.779670 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:49.779686 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:52:49.779742 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:49.780194 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:49.780199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:49.780204 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 00:52:49.780458 (+ 254us) client_negotiation.cc:240] Negotiation successful
0809 00:52:49.780514 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":568,"threads_started":1}
I20260809 00:52:49.781330 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.776812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:39241 (local address 127.9.225.1:42567)
0809 00:52:49.777927 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:49.777946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:49.778094 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:49.778501 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:49.778506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:49.778531 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:49.779468 (+ 937us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:52:49.779470 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:49.779486 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:52:49.779529 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:49.779933 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:49.779939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:49.779944 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 00:52:49.780119 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 00:52:49.780188 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":617,"threads_started":1}
I20260809 00:52:49.782648 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c7c4d15b764610a6f1485eef9fecab" candidate_uuid: "e4e2a259a353437599c2cd007b56e596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f056cb36721414ca7eaaf6e301d9500" is_pre_election: true
I20260809 00:52:49.782649 18119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c7c4d15b764610a6f1485eef9fecab" candidate_uuid: "e4e2a259a353437599c2cd007b56e596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2715406f80c4f35ab51dd9575106e2b" is_pre_election: true
I20260809 00:52:49.783324 18314 raft_consensus.cc:2454] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4e2a259a353437599c2cd007b56e596 in term 0.
I20260809 00:52:49.783324 18119 raft_consensus.cc:2454] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4e2a259a353437599c2cd007b56e596 in term 0.
I20260809 00:52:49.784264 17828 leader_election.cc:302] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f056cb36721414ca7eaaf6e301d9500, e4e2a259a353437599c2cd007b56e596; no voters:
I20260809 00:52:49.784844 18425 raft_consensus.cc:2781] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:52:49.785028 18425 raft_consensus.cc:482] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:52:49.785151 18425 raft_consensus.cc:3037] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:49.787583 18425 raft_consensus.cc:504] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2715406f80c4f35ab51dd9575106e2b" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39241 } } peers { permanent_uuid: "e4e2a259a353437599c2cd007b56e596" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 41021 } } peers { permanent_uuid: "5f056cb36721414ca7eaaf6e301d9500" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46717 } }
I20260809 00:52:49.788506 18425 leader_election.cc:288] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [CANDIDATE]: Term 1 election: Requested vote from peers c2715406f80c4f35ab51dd9575106e2b (127.9.225.2:39241), 5f056cb36721414ca7eaaf6e301d9500 (127.9.225.3:46717)
I20260809 00:52:49.789173 18119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c7c4d15b764610a6f1485eef9fecab" candidate_uuid: "e4e2a259a353437599c2cd007b56e596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2715406f80c4f35ab51dd9575106e2b"
I20260809 00:52:49.789193 18314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81c7c4d15b764610a6f1485eef9fecab" candidate_uuid: "e4e2a259a353437599c2cd007b56e596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f056cb36721414ca7eaaf6e301d9500"
I20260809 00:52:49.789471 18119 raft_consensus.cc:3037] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:49.789489 18314 raft_consensus.cc:3037] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:49.792060 18314 raft_consensus.cc:2454] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4e2a259a353437599c2cd007b56e596 in term 1.
I20260809 00:52:49.792138 18119 raft_consensus.cc:2454] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4e2a259a353437599c2cd007b56e596 in term 1.
I20260809 00:52:49.792745 17828 leader_election.cc:302] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5f056cb36721414ca7eaaf6e301d9500, c2715406f80c4f35ab51dd9575106e2b, e4e2a259a353437599c2cd007b56e596; no voters:
I20260809 00:52:49.793154 18425 raft_consensus.cc:2781] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:49.793756 18425 raft_consensus.cc:686] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [term 1 LEADER]: Becoming Leader. State: Replica: e4e2a259a353437599c2cd007b56e596, State: Running, Role: LEADER
I20260809 00:52:49.794330 18425 consensus_queue.cc:237] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2715406f80c4f35ab51dd9575106e2b" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39241 } } peers { permanent_uuid: "e4e2a259a353437599c2cd007b56e596" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 41021 } } peers { permanent_uuid: "5f056cb36721414ca7eaaf6e301d9500" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46717 } }
I20260809 00:52:49.799944 17705 catalog_manager.cc:5495] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4e2a259a353437599c2cd007b56e596 (127.9.225.1). New cstate: current_term: 1 leader_uuid: "e4e2a259a353437599c2cd007b56e596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2715406f80c4f35ab51dd9575106e2b" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4e2a259a353437599c2cd007b56e596" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f056cb36721414ca7eaaf6e301d9500" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46717 } health_report { overall_health: UNKNOWN } } }
I20260809 00:52:49.850256 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.847673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.9.225.1:41021)
0809 00:52:49.847912 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:49.847920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:49.847942 (+ 22us) server_negotiation.cc:407] Connection header received
0809 00:52:49.848219 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:49.848223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:49.848330 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:49.848453 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:49.849574 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:49.849578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:49.849581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:49.849599 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:49.849633 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:49.849646 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:49.849796 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:49.849900 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 00:52:49.849970 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 00:52:49.860148 18119 raft_consensus.cc:1261] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b [term 1 FOLLOWER]: Refusing update from remote peer e4e2a259a353437599c2cd007b56e596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:49.860150 18314 raft_consensus.cc:1261] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500 [term 1 FOLLOWER]: Refusing update from remote peer e4e2a259a353437599c2cd007b56e596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:52:49.861400 18435 consensus_queue.cc:1035] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f056cb36721414ca7eaaf6e301d9500" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:49.861860 18425 consensus_queue.cc:1035] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2715406f80c4f35ab51dd9575106e2b" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 39241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:52:49.877898 18438 mvcc.cc:205] Tried to move back new op lower bound from 7316425809334493184 to 7316425809082089472. Current Snapshot: MvccSnapshot[applied={T|T < 7316425809334493184}]
I20260809 00:52:49.897233 17705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:49.897650 17705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:52:49.902837 17705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d555e466bf394e11a094923e45ca0299: Sending DeleteTablet for 3 replicas of tablet 81c7c4d15b764610a6f1485eef9fecab
I20260809 00:52:49.904372 18086 tablet_service.cc:1514] Processing DeleteTablet for tablet 81c7c4d15b764610a6f1485eef9fecab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:49 UTC) from {username='slave'} at 127.0.0.1:54142
I20260809 00:52:49.904387 17894 tablet_service.cc:1514] Processing DeleteTablet for tablet 81c7c4d15b764610a6f1485eef9fecab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:49 UTC) from {username='slave'} at 127.0.0.1:56750
I20260809 00:52:49.904482 18280 tablet_service.cc:1514] Processing DeleteTablet for tablet 81c7c4d15b764610a6f1485eef9fecab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:52:49 UTC) from {username='slave'} at 127.0.0.1:33782
I20260809 00:52:49.907620 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 17802
I20260809 00:52:49.921713 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:49.909653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.9.225.1:41021)
0809 00:52:49.909914 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:49.909924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:49.914034 (+ 4110us) server_negotiation.cc:407] Connection header received
0809 00:52:49.914456 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:49.914467 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:49.914565 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:49.914668 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:49.920026 (+ 5358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:49.920039 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:49.920047 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:49.920089 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:49.920165 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:49.920191 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:49.920392 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:49.921036 (+ 644us) server_negotiation.cc:299] Negotiation successful
0809 00:52:49.921147 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 00:52:49.923799 17983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:49.925114 18457 tablet_replica.cc:330] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500: stopping tablet replica
I20260809 00:52:49.925846 18457 raft_consensus.cc:2229] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:49.926605 18457 raft_consensus.cc:2258] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:49.928918 18454 tablet_replica.cc:330] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b: stopping tablet replica
I20260809 00:52:49.929805 18454 raft_consensus.cc:2229] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:52:49.930104 18457 ts_tablet_manager.cc:1905] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:49.930717 18454 raft_consensus.cc:2258] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:52:49.934587 18454 ts_tablet_manager.cc:1905] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:52:49.938158 18457 ts_tablet_manager.cc:1918] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:52:49.938402 18457 log.cc:1196] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/81c7c4d15b764610a6f1485eef9fecab
I20260809 00:52:49.939046 18457 ts_tablet_manager.cc:1939] T 81c7c4d15b764610a6f1485eef9fecab P 5f056cb36721414ca7eaaf6e301d9500: Deleting consensus metadata
I20260809 00:52:49.941421 17687 catalog_manager.cc:4841] TS 5f056cb36721414ca7eaaf6e301d9500 (127.9.225.3:46717): tablet 81c7c4d15b764610a6f1485eef9fecab (table test-table [id=d761835f22004be9b0011c0b5953f53c]) successfully deleted
I20260809 00:52:49.942454 18454 ts_tablet_manager.cc:1918] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:52:49.942651 18454 log.cc:1196] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/81c7c4d15b764610a6f1485eef9fecab
I20260809 00:52:49.943167 18454 ts_tablet_manager.cc:1939] T 81c7c4d15b764610a6f1485eef9fecab P c2715406f80c4f35ab51dd9575106e2b: Deleting consensus metadata
I20260809 00:52:49.944940 17683 catalog_manager.cc:4841] TS c2715406f80c4f35ab51dd9575106e2b (127.9.225.2:39241): tablet 81c7c4d15b764610a6f1485eef9fecab (table test-table [id=d761835f22004be9b0011c0b5953f53c]) successfully deleted
I20260809 00:52:50.036897 18456 tablet_replica.cc:330] T 81c7c4d15b764610a6f1485eef9fecab P e4e2a259a353437599c2cd007b56e596: stopping tablet replica
I20260809 00:52:50.036963 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 17802
I20260809 00:52:50.070425 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 17996
W20260809 00:52:50.070451 17687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.225.1:41021 (error 108)
I20260809 00:52:50.072642 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:50.072145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:41021 (local address 127.0.0.1:56776)
0809 00:52:50.072368 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:50.072494 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260809 00:52:50.072999 17687 catalog_manager.cc:4584] TS e4e2a259a353437599c2cd007b56e596 (127.9.225.1:41021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81c7c4d15b764610a6f1485eef9fecab: Network error: Client connection negotiation failed: client connection to 127.9.225.1:41021: connect: Connection refused (error 111)
I20260809 00:52:50.074118 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:50.071443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.9.225.2:39241)
0809 00:52:50.071740 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:50.071745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:50.071767 (+ 22us) server_negotiation.cc:407] Connection header received
0809 00:52:50.071925 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:50.071930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:50.072022 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:50.072116 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:50.073411 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:50.073420 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:50.073424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:50.073451 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:50.073487 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:50.073501 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:50.073615 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:50.073748 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 00:52:50.073801 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 00:52:50.074879 18176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:50.127024 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:50.126336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:41021 (local address 127.0.0.1:56792)
0809 00:52:50.126693 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:50.126815 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260809 00:52:50.128321 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:50.127873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:41021 (local address 127.0.0.1:56798)
0809 00:52:50.128058 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:50.128166 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 00:52:50.169402 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 17996
I20260809 00:52:50.189291 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:50.188715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:41021 (local address 127.0.0.1:56814)
0809 00:52:50.188980 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:50.189094 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260809 00:52:50.190547 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:50.190177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:41021 (local address 127.0.0.1:56822)
0809 00:52:50.190339 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:50.190427 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 00:52:50.198208 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 18188
I20260809 00:52:50.201103 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:50.199132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.9.225.3:46717)
0809 00:52:50.199316 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:50.199321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:50.199342 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:52:50.199408 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:50.199412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:50.199520 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:50.199608 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:50.200528 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:50.200533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:50.200536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:50.200553 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:50.200587 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:50.200599 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:50.200715 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:50.200808 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 00:52:50.200862 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 00:52:50.201861 18376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:50.271472 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:50.270878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:41021 (local address 127.0.0.1:56828)
0809 00:52:50.271187 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:50.271291 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260809 00:52:50.272709 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:50.272266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:41021 (local address 127.0.0.1:56832)
0809 00:52:50.272506 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:50.272578 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:41021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260809 00:52:50.292963 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 18188
I20260809 00:52:50.318990 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 17661
I20260809 00:52:50.320399 17760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:50.434671 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 17661
2026-08-09T00:52:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3135 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 00:52:50.489454 10116 mini_oidc.cc:152] Starting JWKS server
I20260809 00:52:50.515728 10116 webserver.cc:460] Webserver started at https://127.0.0.1:39497/ using document root <none> and password file <none>
I20260809 00:52:50.516227 10116 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 00:52:50.517040 10116 webserver.cc:460] Webserver started at http://127.0.0.1:34067/ using document root <none> and password file <none>
2026-08-09T00:52:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:50Z Disabled control of system clock
I20260809 00:52:50.557585 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39189
--webserver_interface=127.9.225.62
--webserver_port=0
--builtin_ntp_servers=127.9.225.20:36775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:39189
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39497/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/testchainca.cert with env {}
W20260809 00:52:50.939656 18473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:50.940099 18473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:50.940276 18473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:50.948109 18473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:50.948314 18473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:50.948414 18473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:50.948518 18473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:50.948601 18473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 00:52:50.948698 18473 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 00:52:50.948781 18473 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39497/jwks.json
I20260809 00:52:50.960246 18473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:39189
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39497/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:50.961980 18473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:50.963495 18473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:50.977252 18484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:50.977411 18481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:50.977728 18480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:50.978679 18473 server_base.cc:1034] running on GCE node
I20260809 00:52:50.979678 18473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:50.983839 18473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:50.985188 18473 hybrid_clock.cc:648] HybridClock initialized: now 1786236770985103 us; error 101 us; skew 500 ppm
I20260809 00:52:50.985711 18473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:52:50.988551 18473 webserver.cc:460] Webserver started at http://127.9.225.62:38351/ using document root <none> and password file <none>
I20260809 00:52:50.989457 18473 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:50.989614 18473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:50.989993 18473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:50.993531 18473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "0e3a9a1a35cb44e59f168ee70b375263"
format_stamp: "Formatted at 2026-08-09 00:52:50 on dist-test-slave-j2vl"
server_key: "e46447b4f880846686f750f31b7c2a2c"
server_key_iv: "63bc6dcd46b9413b42bfdb2d4ac02e62"
server_key_version: "encryptionkey@0"
I20260809 00:52:50.994541 18473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "0e3a9a1a35cb44e59f168ee70b375263"
format_stamp: "Formatted at 2026-08-09 00:52:50 on dist-test-slave-j2vl"
server_key: "e46447b4f880846686f750f31b7c2a2c"
server_key_iv: "63bc6dcd46b9413b42bfdb2d4ac02e62"
server_key_version: "encryptionkey@0"
I20260809 00:52:51.001065 18473 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 00:52:51.005275 18492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:51.007102 18473 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:52:51.007364 18473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "0e3a9a1a35cb44e59f168ee70b375263"
format_stamp: "Formatted at 2026-08-09 00:52:50 on dist-test-slave-j2vl"
server_key: "e46447b4f880846686f750f31b7c2a2c"
server_key_iv: "63bc6dcd46b9413b42bfdb2d4ac02e62"
server_key_version: "encryptionkey@0"
I20260809 00:52:51.007652 18473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:51.023159 18473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:51.024178 18473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:51.024544 18473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:51.045321 18473 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:39189
I20260809 00:52:51.045392 18589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:39189 every 8 connection(s)
I20260809 00:52:51.047080 18473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:51.051529 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:51.056170 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 18473
I20260809 00:52:51.056432 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:51.057277 10116 external_mini_cluster.cc:1428] Setting key ce4e6d9ed2aaae4cacdd7ad931560006
I20260809 00:52:51.067487 18590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263: Bootstrap starting.
I20260809 00:52:51.072000 18590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:51.073460 18590 log.cc:824] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:51.073868 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:51.059601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.9.225.62:39189)
0809 00:52:51.060755 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:51.060769 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:51.060818 (+ 49us) server_negotiation.cc:407] Connection header received
0809 00:52:51.062727 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:51.062770 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:51.063411 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:51.064109 (+ 698us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:51.064951 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:51.066524 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:51.068301 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:51.069231 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:51.069538 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:51.071019 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:51.071029 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:51.071047 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:51.071673 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:51.071778 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:51.071804 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:51.071958 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:51.072248 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 00:52:51.072639 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":401,"threads_started":1}
I20260809 00:52:51.077135 18590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263: No bootstrap required, opened a new log
I20260809 00:52:51.078364 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:52:51.085737 18590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3a9a1a35cb44e59f168ee70b375263" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39189 } }
I20260809 00:52:51.086228 18590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:51.086351 18590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3a9a1a35cb44e59f168ee70b375263, State: Initialized, Role: FOLLOWER
I20260809 00:52:51.087064 18590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3a9a1a35cb44e59f168ee70b375263" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39189 } }
I20260809 00:52:51.087339 18590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:51.087504 18590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:51.087723 18590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:51.090544 18590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3a9a1a35cb44e59f168ee70b375263" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39189 } }
I20260809 00:52:51.091144 18590 leader_election.cc:302] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e3a9a1a35cb44e59f168ee70b375263; no voters:
I20260809 00:52:51.091777 18590 leader_election.cc:288] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:51.091971 18596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:51.092752 18596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [term 1 LEADER]: Becoming Leader. State: Replica: 0e3a9a1a35cb44e59f168ee70b375263, State: Running, Role: LEADER
I20260809 00:52:51.093511 18596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3a9a1a35cb44e59f168ee70b375263" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39189 } }
I20260809 00:52:51.094442 18590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:51.097199 18598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e3a9a1a35cb44e59f168ee70b375263. Latest consensus state: current_term: 1 leader_uuid: "0e3a9a1a35cb44e59f168ee70b375263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3a9a1a35cb44e59f168ee70b375263" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39189 } } }
I20260809 00:52:51.097174 18597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e3a9a1a35cb44e59f168ee70b375263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e3a9a1a35cb44e59f168ee70b375263" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39189 } } }
I20260809 00:52:51.097805 18598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:51.097950 18597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:51.102934 18606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:51.109135 18606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:51.119876 18606 catalog_manager.cc:1347] Generated new cluster ID: 71ba57fd1df94960a5973e80d0e99830
I20260809 00:52:51.120028 18606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:51.144430 18606 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:51.146003 18606 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:51.156068 18606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e3a9a1a35cb44e59f168ee70b375263: Generated new TSK 0
I20260809 00:52:51.157183 18606 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 00:52:51.202795 18629 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 00:52:51.203030 18629 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 00:52:51.209982 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:51.180068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.9.225.62:39189)
0809 00:52:51.180287 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:51.180294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:51.180765 (+ 471us) server_negotiation.cc:407] Connection header received
0809 00:52:51.181169 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:51.181175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:51.181288 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:51.181501 (+ 213us) server_negotiation.cc:226] Negotiated authn=JWT
0809 00:52:51.182160 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:51.183339 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:51.184893 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:51.185687 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:51.185841 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:51.186587 (+ 746us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 00:52:51.209015 (+ 22428us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39497/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 00:52:51.209639 (+ 624us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39497/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":85}
W20260809 00:52:51.210155 18594 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39497/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 00:52:51.211905 18628 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:52:51.179868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39189 (local address 127.0.0.1:39388)
0809 00:52:51.180640 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:51.180658 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:51.180849 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:51.181549 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:51.181553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:51.181565 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0809 00:52:51.181969 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:51.181979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:51.183526 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:51.183530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:51.184697 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:51.184709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:51.185932 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:51.185938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:51.186087 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:51.186185 (+ 98us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 00:52:51.209546 (+ 23361us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39497/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 00:52:51.209939 (+ 393us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.225.62:39189: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39497/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":413,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.225.62:39189: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39497/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 00:52:51.214105 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 18473
I20260809 00:52:51.215461 18575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:51.316303 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 18473
2026-08-09T00:52:51Z chronyd exiting
I20260809 00:52:51.375833 10116 test_util.cc:175] -----------------------------------------------
I20260809 00:52:51.376015 10116 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786236719337175-10116-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (892 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 00:52:51.380753 10116 mini_oidc.cc:152] Starting JWKS server
I20260809 00:52:51.383163 10116 webserver.cc:460] Webserver started at https://127.0.0.1:36313/ using document root <none> and password file <none>
I20260809 00:52:51.383514 10116 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 00:52:51.384394 10116 webserver.cc:460] Webserver started at http://127.0.0.1:35407/ using document root <none> and password file <none>
2026-08-09T00:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:51Z Disabled control of system clock
I20260809 00:52:51.425916 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39991
--webserver_interface=127.9.225.62
--webserver_port=0
--builtin_ntp_servers=127.9.225.20:41763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:39991
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36313/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/testchainca.cert with env {}
W20260809 00:52:51.800029 18642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:51.800469 18642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:51.800657 18642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:51.808473 18642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:51.808682 18642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:51.808758 18642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:51.808856 18642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:51.808924 18642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 00:52:51.809021 18642 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 00:52:51.809106 18642 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36313/jwks.json
I20260809 00:52:51.819927 18642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:39991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36313/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:51.822188 18642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:51.824172 18642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:51.837759 18648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:51.837900 18647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:51.837877 18652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:51.838689 18642 server_base.cc:1034] running on GCE node
I20260809 00:52:51.839686 18642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:51.841516 18642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:51.842772 18642 hybrid_clock.cc:648] HybridClock initialized: now 1786236771842723 us; error 53 us; skew 500 ppm
I20260809 00:52:51.843182 18642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:52:51.850514 18642 webserver.cc:460] Webserver started at http://127.9.225.62:35905/ using document root <none> and password file <none>
I20260809 00:52:51.851805 18642 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:51.852028 18642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:51.852525 18642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:51.856930 18642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "7b7c51bf5f4f4069bc5c96fcdb00213d"
format_stamp: "Formatted at 2026-08-09 00:52:51 on dist-test-slave-j2vl"
server_key: "77767108466952b1ecd200445efc608c"
server_key_iv: "6bf10cee3659eb7e2e65e3acf7ca0ac1"
server_key_version: "encryptionkey@0"
I20260809 00:52:51.858470 18642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "7b7c51bf5f4f4069bc5c96fcdb00213d"
format_stamp: "Formatted at 2026-08-09 00:52:51 on dist-test-slave-j2vl"
server_key: "77767108466952b1ecd200445efc608c"
server_key_iv: "6bf10cee3659eb7e2e65e3acf7ca0ac1"
server_key_version: "encryptionkey@0"
I20260809 00:52:51.866461 18642 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260809 00:52:51.872301 18660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:51.874696 18642 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260809 00:52:51.874984 18642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "7b7c51bf5f4f4069bc5c96fcdb00213d"
format_stamp: "Formatted at 2026-08-09 00:52:51 on dist-test-slave-j2vl"
server_key: "77767108466952b1ecd200445efc608c"
server_key_iv: "6bf10cee3659eb7e2e65e3acf7ca0ac1"
server_key_version: "encryptionkey@0"
I20260809 00:52:51.875279 18642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:51.902175 18642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:51.903239 18642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:51.903604 18642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:51.924012 18642 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:39991
I20260809 00:52:51.924017 18736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:39991 every 8 connection(s)
I20260809 00:52:51.925802 18642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:51.930619 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:51.935001 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 18642
I20260809 00:52:51.935220 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:51.935896 10116 external_mini_cluster.cc:1428] Setting key 5d5c5b226c43789bc6f82a6e74d64aa6
I20260809 00:52:51.945487 18737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d: Bootstrap starting.
I20260809 00:52:51.950577 18737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:51.952255 18740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:51.938170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.9.225.62:39991)
0809 00:52:51.939345 (+ 1175us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:51.939357 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:51.939404 (+ 47us) server_negotiation.cc:407] Connection header received
0809 00:52:51.941116 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:51.941151 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:51.941832 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:51.942335 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:51.943008 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:51.944343 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:51.945832 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:51.947060 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:51.947314 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:51.948519 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:51.948532 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:51.948557 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:51.949458 (+ 901us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:51.949606 (+ 148us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:51.949640 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:51.949772 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:51.950063 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 00:52:51.950518 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":410,"threads_started":1}
I20260809 00:52:51.952322 18737 log.cc:824] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:51.956277 18737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d: No bootstrap required, opened a new log
I20260809 00:52:51.957002 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:52:51.964927 18737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7c51bf5f4f4069bc5c96fcdb00213d" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39991 } }
I20260809 00:52:51.965454 18737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:51.965593 18737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7c51bf5f4f4069bc5c96fcdb00213d, State: Initialized, Role: FOLLOWER
I20260809 00:52:51.966567 18737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7c51bf5f4f4069bc5c96fcdb00213d" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39991 } }
I20260809 00:52:51.966976 18737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:51.967146 18737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:51.967376 18737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:51.970237 18737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7c51bf5f4f4069bc5c96fcdb00213d" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39991 } }
I20260809 00:52:51.971010 18737 leader_election.cc:302] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b7c51bf5f4f4069bc5c96fcdb00213d; no voters:
I20260809 00:52:51.971781 18737 leader_election.cc:288] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:51.972030 18745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:51.973385 18745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [term 1 LEADER]: Becoming Leader. State: Replica: 7b7c51bf5f4f4069bc5c96fcdb00213d, State: Running, Role: LEADER
I20260809 00:52:51.974009 18745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7c51bf5f4f4069bc5c96fcdb00213d" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39991 } }
I20260809 00:52:51.974677 18737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:51.977291 18747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b7c51bf5f4f4069bc5c96fcdb00213d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7c51bf5f4f4069bc5c96fcdb00213d" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39991 } } }
I20260809 00:52:51.977774 18747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:51.977480 18748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b7c51bf5f4f4069bc5c96fcdb00213d. Latest consensus state: current_term: 1 leader_uuid: "7b7c51bf5f4f4069bc5c96fcdb00213d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7c51bf5f4f4069bc5c96fcdb00213d" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39991 } } }
I20260809 00:52:51.978025 18748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:51.980095 18754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:51.987077 18754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:51.998652 18754 catalog_manager.cc:1347] Generated new cluster ID: 273d97c240714824b4533b3b2d50b32f
I20260809 00:52:51.998816 18754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:52.018183 18754 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:52.020179 18754 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:52.036407 18754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b7c51bf5f4f4069bc5c96fcdb00213d: Generated new TSK 0
I20260809 00:52:52.037709 18754 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 00:52:52.073786 18783 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 00:52:52.073945 18783 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 00:52:52.075567 18740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:52.053623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58752 (local address 127.9.225.62:39991)
0809 00:52:52.053867 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:52.053878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:52.054421 (+ 543us) server_negotiation.cc:407] Connection header received
0809 00:52:52.054737 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:52.054743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:52.054842 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:52.055031 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0809 00:52:52.055677 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:52.056460 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:52.058136 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:52.058620 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:52.058789 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:52.059405 (+ 616us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 00:52:52.074628 (+ 15223us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36313/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 00:52:52.075188 (+ 560us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36313/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260809 00:52:52.075727 18740 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58752: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36313/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260809 00:52:52.075814 18779 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:52:52.053447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39991 (local address 127.0.0.1:58752)
0809 00:52:52.054281 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:52.054300 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:52.054508 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:52.055060 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:52.055065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:52.055077 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0809 00:52:52.055465 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:52.055477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:52.056656 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:52.056661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:52.057870 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:52.057890 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:52.058840 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:52.058847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:52.059023 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:52.059065 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 00:52:52.075001 (+ 15936us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36313/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 00:52:52.075312 (+ 311us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.225.62:39991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36313/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":505,"threads_started":1}
I20260809 00:52:52.077873 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 18642
I20260809 00:52:52.079957 18732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:52.180318 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 18642
2026-08-09T00:52:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (866 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 00:52:52.246600 10116 mini_oidc.cc:152] Starting JWKS server
I20260809 00:52:52.248631 10116 webserver.cc:460] Webserver started at https://127.0.0.1:40267/ using document root <none> and password file <none>
I20260809 00:52:52.248870 10116 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 00:52:52.249701 10116 webserver.cc:460] Webserver started at http://127.0.0.1:44379/ using document root <none> and password file <none>
2026-08-09T00:52:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:52Z Disabled control of system clock
I20260809 00:52:52.291288 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43247
--webserver_interface=127.9.225.62
--webserver_port=0
--builtin_ntp_servers=127.9.225.20:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:43247
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40267/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 00:52:52.683944 18792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:52.684361 18792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:52.684545 18792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:52.692308 18792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:52.692520 18792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:52.692632 18792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:52.692746 18792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:52.692840 18792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 00:52:52.692937 18792 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 00:52:52.693027 18792 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40267/jwks.json
I20260809 00:52:52.703154 18792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:43247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40267/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:52.704773 18792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:52.706943 18792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:52.720597 18800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:52.720656 18805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:52.720733 18802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:52.721977 18792 server_base.cc:1034] running on GCE node
I20260809 00:52:52.723091 18792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:52.724982 18792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:52.726260 18792 hybrid_clock.cc:648] HybridClock initialized: now 1786236772726221 us; error 71 us; skew 500 ppm
I20260809 00:52:52.726706 18792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:52:52.729601 18792 webserver.cc:460] Webserver started at http://127.9.225.62:37683/ using document root <none> and password file <none>
I20260809 00:52:52.730458 18792 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:52.730612 18792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:52.730996 18792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:52.734359 18792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "47bb35856f51409981ddebe26c6ff906"
format_stamp: "Formatted at 2026-08-09 00:52:52 on dist-test-slave-j2vl"
server_key: "487e7f7cbccd1ec1d4e8ef5f89c6de2b"
server_key_iv: "97deda57379743e845d98deb938f01c1"
server_key_version: "encryptionkey@0"
I20260809 00:52:52.735322 18792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "47bb35856f51409981ddebe26c6ff906"
format_stamp: "Formatted at 2026-08-09 00:52:52 on dist-test-slave-j2vl"
server_key: "487e7f7cbccd1ec1d4e8ef5f89c6de2b"
server_key_iv: "97deda57379743e845d98deb938f01c1"
server_key_version: "encryptionkey@0"
I20260809 00:52:52.741602 18792 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 00:52:52.745661 18814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:52.747336 18792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 00:52:52.747593 18792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "47bb35856f51409981ddebe26c6ff906"
format_stamp: "Formatted at 2026-08-09 00:52:52 on dist-test-slave-j2vl"
server_key: "487e7f7cbccd1ec1d4e8ef5f89c6de2b"
server_key_iv: "97deda57379743e845d98deb938f01c1"
server_key_version: "encryptionkey@0"
I20260809 00:52:52.747870 18792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:52.771413 18792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:52.772492 18792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:52.772868 18792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:52.794374 18792 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:43247
I20260809 00:52:52.794407 18896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:43247 every 8 connection(s)
I20260809 00:52:52.796053 18792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:52.800621 18897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:52.800956 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 18792
I20260809 00:52:52.801182 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:52.801887 10116 external_mini_cluster.cc:1428] Setting key 6254555696e734ebfec2c575a3ecf401
I20260809 00:52:52.815375 18897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906: Bootstrap starting.
I20260809 00:52:52.817983 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:52.803929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.9.225.62:43247)
0809 00:52:52.805314 (+ 1385us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:52.805332 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:52.805446 (+ 114us) server_negotiation.cc:407] Connection header received
0809 00:52:52.808127 (+ 2681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:52.808160 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:52.808773 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:52.809249 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:52.809908 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:52.811198 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:52.812640 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:52.813533 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:52.813753 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:52.815167 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:52.815177 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:52.815196 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:52.815916 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:52.816025 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:52.816051 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:52.816193 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:52.816461 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 00:52:52.816832 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":462,"threads_started":1}
I20260809 00:52:52.819362 18897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:52.820688 18897 log.cc:824] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:52.822511 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:52:52.824169 18897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906: No bootstrap required, opened a new log
I20260809 00:52:52.834151 18897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bb35856f51409981ddebe26c6ff906" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43247 } }
I20260809 00:52:52.834613 18897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:52.834699 18897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47bb35856f51409981ddebe26c6ff906, State: Initialized, Role: FOLLOWER
I20260809 00:52:52.835352 18897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bb35856f51409981ddebe26c6ff906" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43247 } }
I20260809 00:52:52.835582 18897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:52.835686 18897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:52.835839 18897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:52.838308 18897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bb35856f51409981ddebe26c6ff906" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43247 } }
I20260809 00:52:52.839107 18897 leader_election.cc:302] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47bb35856f51409981ddebe26c6ff906; no voters:
I20260809 00:52:52.839905 18897 leader_election.cc:288] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:52.840011 18904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:52.840829 18904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [term 1 LEADER]: Becoming Leader. State: Replica: 47bb35856f51409981ddebe26c6ff906, State: Running, Role: LEADER
I20260809 00:52:52.841486 18904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bb35856f51409981ddebe26c6ff906" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43247 } }
I20260809 00:52:52.842355 18897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:52.844623 18909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47bb35856f51409981ddebe26c6ff906. Latest consensus state: current_term: 1 leader_uuid: "47bb35856f51409981ddebe26c6ff906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bb35856f51409981ddebe26c6ff906" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43247 } } }
I20260809 00:52:52.844851 18907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47bb35856f51409981ddebe26c6ff906" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bb35856f51409981ddebe26c6ff906" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43247 } } }
I20260809 00:52:52.845307 18907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:52.845307 18909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:52.850135 18916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:52.856571 18916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:52.866707 18916 catalog_manager.cc:1347] Generated new cluster ID: 381b72dd94fa44fe8e85b848be88089b
I20260809 00:52:52.866879 18916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:52.875865 18916 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:52.878033 18916 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:52.902319 18916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47bb35856f51409981ddebe26c6ff906: Generated new TSK 0
I20260809 00:52:52.904155 18916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:52:52.940959 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:52.930763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.9.225.62:43247)
0809 00:52:52.931023 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:52.931031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:52.931412 (+ 381us) server_negotiation.cc:407] Connection header received
0809 00:52:52.931751 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:52.931756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:52.931842 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:52.932026 (+ 184us) server_negotiation.cc:226] Negotiated authn=JWT
0809 00:52:52.932710 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:52.933625 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:52.935878 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:52.936708 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:52.936825 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:52.937647 (+ 822us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 00:52:52.940200 (+ 2553us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40267/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 00:52:52.940601 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48892: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40267/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":102}
W20260809 00:52:52.941205 18902 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48892: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40267/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 00:52:52.943277 18946 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:52:52.930542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43247 (local address 127.0.0.1:48892)
0809 00:52:52.931270 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:52.931303 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:52.931484 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:52.932075 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:52.932081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:52.932097 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0809 00:52:52.932442 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:52.932459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:52.933801 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:52.933808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:52.935679 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:52.935699 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:52.936963 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:52.936972 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:52.937187 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:52.937242 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 00:52:52.940475 (+ 3233us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40267/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 00:52:52.942747 (+ 2272us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.225.62:43247: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40267/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":1864,"thread_start_us":389,"threads_started":1}
I20260809 00:52:52.944476 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 18792
I20260809 00:52:52.946460 18890 generic_service.cc:196] Checking for leaks (request via RPC)
E20260809 00:52:52.955703 18948 webserver.cc:533] Webserver: sslize failed:
E20260809 00:52:52.955901 18948 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 00:52:53.046811 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 18792
2026-08-09T00:52:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (872 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T00:52:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:53Z Disabled control of system clock
I20260809 00:52:53.160544 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:44693
--webserver_interface=127.9.225.62
--webserver_port=0
--builtin_ntp_servers=127.9.225.20:43309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:44693
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/testchainca.cert with env {}
W20260809 00:52:53.541818 18954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:53.542266 18954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:53.542363 18954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:53.550056 18954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:53.550180 18954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:53.550225 18954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:53.550275 18954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:53.550321 18954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 00:52:53.550379 18954 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 00:52:53.550420 18954 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 00:52:53.560947 18954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:44693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:44693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:53.562685 18954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:53.564241 18954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:53.578538 18966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:53.578781 18961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:53.578790 18962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:53.580127 18954 server_base.cc:1034] running on GCE node
I20260809 00:52:53.581146 18954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:53.582942 18954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:53.584203 18954 hybrid_clock.cc:648] HybridClock initialized: now 1786236773584138 us; error 68 us; skew 500 ppm
I20260809 00:52:53.584620 18954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:52:53.587401 18954 webserver.cc:460] Webserver started at http://127.9.225.62:41511/ using document root <none> and password file <none>
I20260809 00:52:53.588268 18954 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:53.588425 18954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:53.588780 18954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:53.594034 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "9a1ddacb6c9343febf05e8c2249941ee"
format_stamp: "Formatted at 2026-08-09 00:52:53 on dist-test-slave-j2vl"
server_key: "5d520fffb837ea15036bb1b0efdab455"
server_key_iv: "41288d649137e7eb469b524451ef616f"
server_key_version: "encryptionkey@0"
I20260809 00:52:53.595583 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "9a1ddacb6c9343febf05e8c2249941ee"
format_stamp: "Formatted at 2026-08-09 00:52:53 on dist-test-slave-j2vl"
server_key: "5d520fffb837ea15036bb1b0efdab455"
server_key_iv: "41288d649137e7eb469b524451ef616f"
server_key_version: "encryptionkey@0"
I20260809 00:52:53.603876 18954 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260809 00:52:53.610261 18975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:53.613245 18954 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.004s
I20260809 00:52:53.613763 18954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "9a1ddacb6c9343febf05e8c2249941ee"
format_stamp: "Formatted at 2026-08-09 00:52:53 on dist-test-slave-j2vl"
server_key: "5d520fffb837ea15036bb1b0efdab455"
server_key_iv: "41288d649137e7eb469b524451ef616f"
server_key_version: "encryptionkey@0"
I20260809 00:52:53.614099 18954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:53.644208 18954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:53.645625 18954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:53.646021 18954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:53.670068 18954 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:44693
I20260809 00:52:53.670109 19061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:44693 every 8 connection(s)
I20260809 00:52:53.671650 18954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:53.672282 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 18954
I20260809 00:52:53.672598 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:53.673542 10116 external_mini_cluster.cc:1428] Setting key 777825d5921dc03f29419b9ac5f09e7f
I20260809 00:52:53.677011 19062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:53.688961 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:53.675709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49172 (local address 127.9.225.62:44693)
0809 00:52:53.677018 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:53.677031 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:53.677107 (+ 76us) server_negotiation.cc:407] Connection header received
0809 00:52:53.678950 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:53.678973 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:53.679487 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:53.679867 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:53.680618 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:53.682143 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:53.684079 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:53.684810 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:53.684961 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:53.686353 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:53.686361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:53.686379 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:52:53.686980 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:53.687070 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:52:53.687093 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:53.687193 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:53.687452 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 00:52:53.687854 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":421,"threads_started":1}
I20260809 00:52:53.690758 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee: Bootstrap starting.
I20260809 00:52:53.693490 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:52:53.694805 19062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:53.696275 19062 log.cc:824] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee: Log is configured to *not* fsync() on all Append() calls
I20260809 00:52:53.699687 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee: No bootstrap required, opened a new log
I20260809 00:52:53.707718 19062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1ddacb6c9343febf05e8c2249941ee" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44693 } }
I20260809 00:52:53.708165 19062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:53.708246 19062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1ddacb6c9343febf05e8c2249941ee, State: Initialized, Role: FOLLOWER
I20260809 00:52:53.708811 19062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1ddacb6c9343febf05e8c2249941ee" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44693 } }
I20260809 00:52:53.709046 19062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:53.709159 19062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:53.709304 19062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:53.711585 19062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1ddacb6c9343febf05e8c2249941ee" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44693 } }
I20260809 00:52:53.712113 19062 leader_election.cc:302] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a1ddacb6c9343febf05e8c2249941ee; no voters:
I20260809 00:52:53.712754 19062 leader_election.cc:288] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:53.712973 19067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:53.713847 19067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [term 1 LEADER]: Becoming Leader. State: Replica: 9a1ddacb6c9343febf05e8c2249941ee, State: Running, Role: LEADER
I20260809 00:52:53.714643 19067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1ddacb6c9343febf05e8c2249941ee" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44693 } }
I20260809 00:52:53.715412 19062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:53.717988 19072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a1ddacb6c9343febf05e8c2249941ee. Latest consensus state: current_term: 1 leader_uuid: "9a1ddacb6c9343febf05e8c2249941ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1ddacb6c9343febf05e8c2249941ee" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44693 } } }
I20260809 00:52:53.718395 19072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:53.717941 19069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a1ddacb6c9343febf05e8c2249941ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1ddacb6c9343febf05e8c2249941ee" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44693 } } }
I20260809 00:52:53.718683 19069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:53.721207 19079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:53.727376 19079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:53.738464 19079 catalog_manager.cc:1347] Generated new cluster ID: b2b63e2a8c9444b1b5879e7e7122d0d2
I20260809 00:52:53.738622 19079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:53.772020 19079 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:53.774044 19079 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:53.792182 19079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a1ddacb6c9343febf05e8c2249941ee: Generated new TSK 0
I20260809 00:52:53.793279 19079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:52:54.264772 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:53.816155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.9.225.62:44693)
0809 00:52:53.816386 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:53.816393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:53.816883 (+ 490us) server_negotiation.cc:407] Connection header received
0809 00:52:53.817163 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:53.817170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:53.817280 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:53.817536 (+ 256us) server_negotiation.cc:226] Negotiated authn=JWT
0809 00:52:53.818062 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:53.819275 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:53.820914 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:53.821687 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:53.821843 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:53.822463 (+ 620us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 00:52:54.264133 (+441670us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 00:52:54.264513 (+ 380us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49180: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":88}
W20260809 00:52:54.264936 19065 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49180: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 00:52:54.265211 19098 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:52:53.815981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44693 (local address 127.0.0.1:49180)
0809 00:52:53.816781 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:53.816799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:53.816940 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:53.817545 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:53.817549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:53.817571 (+ 22us) client_negotiation.cc:189] Negotiated authn=JWT
0809 00:52:53.817877 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:53.817888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:53.819457 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:53.819462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:53.820688 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:53.820703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:53.821900 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:53.821907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:53.822079 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:53.822120 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 00:52:54.264485 (+442365us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 00:52:54.264763 (+ 278us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.225.62:44693: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":440,"threads_started":1}
I20260809 00:52:54.266294 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 18954
I20260809 00:52:54.267596 19056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:54.375921 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 18954
2026-08-09T00:52:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1307 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (483 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (474 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:52:55 dist-test-slave-j2vl krb5kdc[19123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:52:55 dist-test-slave-j2vl krb5kdc[19123](info): set up 2 sockets
Aug 09 00:52:55 dist-test-slave-j2vl krb5kdc[19123](info): commencing operation
krb5kdc: starting...
W20260809 00:52:56.726078 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.339s user 0.006s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:52:56 dist-test-slave-j2vl krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:52:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:52:57.018047 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:42201
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:42201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:57.400738 19141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:57.401165 19141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:57.401360 19141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:57.409055 19141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:52:57.409250 19141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:57.409407 19141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:57.409521 19141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:52:57.409605 19141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:52:57.420298 19141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:42201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:42201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:57.421922 19141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:57.423429 19141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:57.437001 19146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:57.436977 19150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:57.436955 19147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:57.437932 19141 server_base.cc:1034] running on GCE node
I20260809 00:52:57.438947 19141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:57.440714 19141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:57.442021 19141 hybrid_clock.cc:648] HybridClock initialized: now 1786236777441935 us; error 67 us; skew 500 ppm
I20260809 00:52:57.442468 19141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:57 dist-test-slave-j2vl krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236777, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:57.452437 19141 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:52:57.454384 19141 webserver.cc:460] Webserver started at http://127.9.225.62:37739/ using document root <none> and password file <none>
I20260809 00:52:57.455241 19141 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:57.455415 19141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:57.455794 19141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:57.459347 19141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "a6f3b814a54e4a18a358dd85511f59e3"
format_stamp: "Formatted at 2026-08-09 00:52:57 on dist-test-slave-j2vl"
server_key: "4597d60873ccb8f5a207e48f171e6810"
server_key_iv: "39efa18f855ebf9610fd8c10f71b4117"
server_key_version: "encryptionkey@0"
I20260809 00:52:57.460279 19141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "a6f3b814a54e4a18a358dd85511f59e3"
format_stamp: "Formatted at 2026-08-09 00:52:57 on dist-test-slave-j2vl"
server_key: "4597d60873ccb8f5a207e48f171e6810"
server_key_iv: "39efa18f855ebf9610fd8c10f71b4117"
server_key_version: "encryptionkey@0"
I20260809 00:52:57.466723 19141 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 00:52:57.471637 19165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:57.473441 19141 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:52:57.473671 19141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "a6f3b814a54e4a18a358dd85511f59e3"
format_stamp: "Formatted at 2026-08-09 00:52:57 on dist-test-slave-j2vl"
server_key: "4597d60873ccb8f5a207e48f171e6810"
server_key_iv: "39efa18f855ebf9610fd8c10f71b4117"
server_key_version: "encryptionkey@0"
I20260809 00:52:57.473937 19141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:57.492864 19141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:57.498006 19141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:57.498348 19141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:57.518904 19141 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:42201
I20260809 00:52:57.518970 19259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:42201 every 8 connection(s)
I20260809 00:52:57.520596 19141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:52:57.525811 19261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:52:57.528410 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 19141
I20260809 00:52:57.528664 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:52:57.529470 10116 external_mini_cluster.cc:1428] Setting key 6fbdfc2259e692df882dcea53d34423a
I20260809 00:52:57.540422 19261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3: Bootstrap starting.
I20260809 00:52:57.544941 19261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3: Neither blocks nor log segments found. Creating new log.
I20260809 00:52:57.546614 19261 log.cc:824] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3: Log is configured to *not* fsync() on all Append() calls
Aug 09 00:52:57 dist-test-slave-j2vl krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:57.550146 19261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3: No bootstrap required, opened a new log
I20260809 00:52:57.556804 19268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:57.532611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.9.225.62:42201)
0809 00:52:57.533737 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:57.533752 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:57.533803 (+ 51us) server_negotiation.cc:407] Connection header received
0809 00:52:57.535717 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:57.535748 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:57.536276 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:57.536715 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:57.538902 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:57.540322 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:57.542224 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:57.543035 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:57.543204 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:57.548678 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:57.548688 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:57.548707 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:57.548748 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:57.552084 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:57.552911 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:57.552922 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:57.552930 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:57.553117 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:57.553638 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:57.553643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:57.553645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:57.554274 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:57.554675 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:57.555108 (+ 433us) server_negotiation.cc:299] Negotiation successful
0809 00:52:57.555603 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":506,"threads_started":1}
I20260809 00:52:57.558632 19261 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f3b814a54e4a18a358dd85511f59e3" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42201 } }
I20260809 00:52:57.559091 19261 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:52:57.559244 19261 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f3b814a54e4a18a358dd85511f59e3, State: Initialized, Role: FOLLOWER
I20260809 00:52:57.559885 19261 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f3b814a54e4a18a358dd85511f59e3" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42201 } }
I20260809 00:52:57.560217 19261 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:52:57.560370 19261 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:52:57.560528 19261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:52:57.562939 19261 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f3b814a54e4a18a358dd85511f59e3" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42201 } }
I20260809 00:52:57.563561 19261 leader_election.cc:302] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6f3b814a54e4a18a358dd85511f59e3; no voters:
I20260809 00:52:57.564218 19261 leader_election.cc:288] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:52:57.564440 19270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:52:57.565873 19270 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [term 1 LEADER]: Becoming Leader. State: Replica: a6f3b814a54e4a18a358dd85511f59e3, State: Running, Role: LEADER
I20260809 00:52:57.566507 19270 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f3b814a54e4a18a358dd85511f59e3" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42201 } }
I20260809 00:52:57.566923 19261 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:52:57.569955 19273 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6f3b814a54e4a18a358dd85511f59e3. Latest consensus state: current_term: 1 leader_uuid: "a6f3b814a54e4a18a358dd85511f59e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f3b814a54e4a18a358dd85511f59e3" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42201 } } }
I20260809 00:52:57.570457 19273 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:57.569932 19272 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6f3b814a54e4a18a358dd85511f59e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f3b814a54e4a18a358dd85511f59e3" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42201 } } }
I20260809 00:52:57.571002 19272 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3 [sys.catalog]: This master's current role is: LEADER
I20260809 00:52:57.572100 19291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:52:57.578341 19291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:52:57.588948 19291 catalog_manager.cc:1347] Generated new cluster ID: 20ff037ac594430e9d07d2132a8e5b5f
I20260809 00:52:57.589116 19291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:52:57.616912 19291 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:52:57.618589 19291 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:52:57.629204 19291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6f3b814a54e4a18a358dd85511f59e3: Generated new TSK 0
I20260809 00:52:57.630671 19291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:52:57.710275 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42201
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:58.107098 19312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:58.107434 19312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:58.107597 19312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:58.116523 19312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:58.116756 19312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:58.116986 19312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:52:58.129428 19312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:58.131354 19312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:58.133139 19312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:58.148272 19322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:58.148252 19323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:58.148197 19327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:58.149190 19312 server_base.cc:1034] running on GCE node
I20260809 00:52:58.150056 19312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:58.151289 19312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:58.152536 19312 hybrid_clock.cc:648] HybridClock initialized: now 1786236778152493 us; error 39 us; skew 500 ppm
I20260809 00:52:58.152943 19312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:58 dist-test-slave-j2vl krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:58.156668 19312 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:52:58.158520 19312 webserver.cc:460] Webserver started at http://127.9.225.1:43649/ using document root <none> and password file <none>
I20260809 00:52:58.159377 19312 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:58.159543 19312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:58.159900 19312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:58.163473 19312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "901b065d589644208198ed0c0be59bcc"
format_stamp: "Formatted at 2026-08-09 00:52:58 on dist-test-slave-j2vl"
server_key: "3ecdda2a62e2129452bf19ae73816e65"
server_key_iv: "8ca5466275917dbc825b71fc8c8aa399"
server_key_version: "encryptionkey@0"
I20260809 00:52:58.164520 19312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "901b065d589644208198ed0c0be59bcc"
format_stamp: "Formatted at 2026-08-09 00:52:58 on dist-test-slave-j2vl"
server_key: "3ecdda2a62e2129452bf19ae73816e65"
server_key_iv: "8ca5466275917dbc825b71fc8c8aa399"
server_key_version: "encryptionkey@0"
I20260809 00:52:58.171651 19312 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260809 00:52:58.176083 19338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:58.177964 19312 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:52:58.178216 19312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "901b065d589644208198ed0c0be59bcc"
format_stamp: "Formatted at 2026-08-09 00:52:58 on dist-test-slave-j2vl"
server_key: "3ecdda2a62e2129452bf19ae73816e65"
server_key_iv: "8ca5466275917dbc825b71fc8c8aa399"
server_key_version: "encryptionkey@0"
I20260809 00:52:58.178504 19312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:58.217494 19312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:58.223014 19312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:58.223302 19312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:58.224404 19312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:58.225975 19312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:58.226063 19312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:58.226154 19312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:58.226199 19312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:58.263729 19312 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:38993
I20260809 00:52:58.263777 19511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:38993 every 8 connection(s)
I20260809 00:52:58.265301 19312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:52:58.273885 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 19312
I20260809 00:52:58.274132 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:52:58.275063 10116 external_mini_cluster.cc:1428] Setting key 14e7f00048c838be7895338459ab444f
Aug 09 00:52:58 dist-test-slave-j2vl krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:58.292650 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:58.268535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:32941 (local address 127.9.225.62:42201)
0809 00:52:58.269024 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:58.269029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:58.270332 (+ 1303us) server_negotiation.cc:407] Connection header received
0809 00:52:58.272566 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:58.272572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:58.272695 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:58.272821 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:58.275505 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:58.276394 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:58.279162 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:58.279672 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:58.279805 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:58.285001 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:58.285008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:58.285011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:58.285056 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:58.288275 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:58.289296 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:58.289306 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:58.289312 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:58.289531 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:58.290542 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:58.290547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:58.290549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:58.290854 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:58.290994 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:58.292148 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0809 00:52:58.292376 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":237,"threads_started":1}
I20260809 00:52:58.294494 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:58.269159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:42201 (local address 127.9.225.1:32941)
0809 00:52:58.270029 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:58.270078 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:58.272198 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:58.273167 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:58.273182 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:58.274096 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:58.275176 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:58.275215 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:58.276602 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:58.276607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:58.278864 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:58.278889 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:58.279980 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:58.279991 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:58.280377 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:58.281244 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:58.281286 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:58.284655 (+ 3369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:58.288532 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:58.288542 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:58.288560 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:58.289112 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:58.289820 (+ 708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:58.289831 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:58.289839 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:58.290375 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:58.291007 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:58.291020 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:58.291607 (+ 587us) client_negotiation.cc:769] Sending connection context
0809 00:52:58.292006 (+ 399us) client_negotiation.cc:240] Negotiation successful
0809 00:52:58.292713 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":366,"threads_started":1}
I20260809 00:52:58.296525 19512 heartbeater.cc:347] Connected to a master server at 127.9.225.62:42201
I20260809 00:52:58.296936 19512 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:58.298247 19512 heartbeater.cc:511] Master 127.9.225.62:42201 requested a full tablet report, sending...
I20260809 00:52:58.301245 19194 ts_manager.cc:194] Registered new tserver with Master: 901b065d589644208198ed0c0be59bcc (127.9.225.1:38993)
I20260809 00:52:58.303001 19194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:32941
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:52:58.342446 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42201
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:52:58.730845 19522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:58.731263 19522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:58.731446 19522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:58.740072 19522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:58.740279 19522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:58.740477 19522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:52:58.751399 19522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:58.753011 19522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:58.754639 19522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:58.770805 19532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:58.770653 19531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:58.771108 19537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:58.771854 19522 server_base.cc:1034] running on GCE node
I20260809 00:52:58.772610 19522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:58.773976 19522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:58.775321 19522 hybrid_clock.cc:648] HybridClock initialized: now 1786236778775263 us; error 48 us; skew 500 ppm
I20260809 00:52:58.775718 19522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:58 dist-test-slave-j2vl krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:58.780028 19522 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:52:58.781922 19522 webserver.cc:460] Webserver started at http://127.9.225.2:33781/ using document root <none> and password file <none>
I20260809 00:52:58.782773 19522 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:58.783033 19522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:58.783437 19522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:58.787066 19522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "3c9049d98c604b11a3d4a61d2aefecf5"
format_stamp: "Formatted at 2026-08-09 00:52:58 on dist-test-slave-j2vl"
server_key: "61e35e8f1ec4b99acd094a4c52c78529"
server_key_iv: "bc341f0844b6e7f8ed1be525487f1b19"
server_key_version: "encryptionkey@0"
I20260809 00:52:58.788000 19522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "3c9049d98c604b11a3d4a61d2aefecf5"
format_stamp: "Formatted at 2026-08-09 00:52:58 on dist-test-slave-j2vl"
server_key: "61e35e8f1ec4b99acd094a4c52c78529"
server_key_iv: "bc341f0844b6e7f8ed1be525487f1b19"
server_key_version: "encryptionkey@0"
I20260809 00:52:58.794442 19522 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260809 00:52:58.798972 19551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:58.800680 19522 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:52:58.800992 19522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "3c9049d98c604b11a3d4a61d2aefecf5"
format_stamp: "Formatted at 2026-08-09 00:52:58 on dist-test-slave-j2vl"
server_key: "61e35e8f1ec4b99acd094a4c52c78529"
server_key_iv: "bc341f0844b6e7f8ed1be525487f1b19"
server_key_version: "encryptionkey@0"
I20260809 00:52:58.801278 19522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:58.815485 19522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:58.820555 19522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:58.820904 19522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:58.822098 19522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:58.823858 19522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:58.823998 19522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:58.824177 19522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:58.824257 19522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:58.863014 19522 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:40727
I20260809 00:52:58.863124 19746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:40727 every 8 connection(s)
I20260809 00:52:58.864614 19522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:52:58.865851 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 19522
I20260809 00:52:58.866052 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:52:58.866706 10116 external_mini_cluster.cc:1428] Setting key 4bc974a534ee93b0e723606678edaf03
Aug 09 00:52:58 dist-test-slave-j2vl krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236778, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:58.895557 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:58.868067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:46497 (local address 127.9.225.62:42201)
0809 00:52:58.868623 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:58.868630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:58.870208 (+ 1578us) server_negotiation.cc:407] Connection header received
0809 00:52:58.873484 (+ 3276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:58.873490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:58.873579 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:58.873686 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:58.876843 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:58.878084 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:58.881071 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:58.881645 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:58.881773 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:58.886624 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:58.886635 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:58.886643 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:58.886701 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:58.891069 (+ 4368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:58.892204 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:58.892214 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:58.892221 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:58.892453 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:58.893080 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:58.893089 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:58.893095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:58.893602 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:58.893798 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:58.894707 (+ 909us) server_negotiation.cc:299] Negotiation successful
0809 00:52:58.895043 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":300,"threads_started":1}
I20260809 00:52:58.896520 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:58.868742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:42201 (local address 127.9.225.2:46497)
0809 00:52:58.869964 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:58.870037 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:58.873014 (+ 2977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:58.874059 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:58.874084 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:58.875311 (+ 1227us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:58.876507 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:58.876547 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:58.878312 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:58.878321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:58.880799 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:58.880826 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:58.881946 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:58.881957 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:58.882348 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:58.883299 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:58.883327 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:58.886248 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:58.891365 (+ 5117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:58.891377 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:58.891397 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:58.892019 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:58.892667 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:58.892671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:58.892674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:58.892933 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:58.893791 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:58.893799 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:58.894234 (+ 435us) client_negotiation.cc:769] Sending connection context
0809 00:52:58.894559 (+ 325us) client_negotiation.cc:240] Negotiation successful
0809 00:52:58.895110 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":596,"threads_started":1}
I20260809 00:52:58.898458 19747 heartbeater.cc:347] Connected to a master server at 127.9.225.62:42201
I20260809 00:52:58.898878 19747 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
I20260809 00:52:58.899804 19747 heartbeater.cc:511] Master 127.9.225.62:42201 requested a full tablet report, sending...
I20260809 00:52:58.902215 19194 ts_manager.cc:194] Registered new tserver with Master: 3c9049d98c604b11a3d4a61d2aefecf5 (127.9.225.2:40727)
I20260809 00:52:58.903322 19194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:46497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:52:58.938004 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42201
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:52:59.306784 19512 heartbeater.cc:503] Master 127.9.225.62:42201 was elected leader, sending a full tablet report...
W20260809 00:52:59.340473 19753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:52:59.340893 19753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:52:59.341076 19753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:52:59.348891 19753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:52:59.349088 19753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:52:59.349295 19753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:52:59.361526 19753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:52:59.363191 19753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:52:59.364786 19753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:52:59.381382 19764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:59.381600 19766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:52:59.381688 19763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:52:59.383030 19753 server_base.cc:1034] running on GCE node
I20260809 00:52:59.383929 19753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:52:59.385182 19753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:52:59.386421 19753 hybrid_clock.cc:648] HybridClock initialized: now 1786236779386366 us; error 56 us; skew 500 ppm
I20260809 00:52:59.386771 19753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:59.390669 19753 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:52:59.392503 19753 webserver.cc:460] Webserver started at http://127.9.225.3:45319/ using document root <none> and password file <none>
I20260809 00:52:59.393330 19753 fs_manager.cc:359] Metadata directory not provided
I20260809 00:52:59.393535 19753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:52:59.393929 19753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:52:59.397645 19753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "f173bf2cb3994fbabd2f0ffae1af3ee7"
format_stamp: "Formatted at 2026-08-09 00:52:59 on dist-test-slave-j2vl"
server_key: "ffd81a820370bbdbed39f1331f72e685"
server_key_iv: "794ce504bb354e70590d058dc421476b"
server_key_version: "encryptionkey@0"
I20260809 00:52:59.398649 19753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "f173bf2cb3994fbabd2f0ffae1af3ee7"
format_stamp: "Formatted at 2026-08-09 00:52:59 on dist-test-slave-j2vl"
server_key: "ffd81a820370bbdbed39f1331f72e685"
server_key_iv: "794ce504bb354e70590d058dc421476b"
server_key_version: "encryptionkey@0"
I20260809 00:52:59.405136 19753 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 00:52:59.410135 19782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:59.411968 19753 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:52:59.412307 19753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "f173bf2cb3994fbabd2f0ffae1af3ee7"
format_stamp: "Formatted at 2026-08-09 00:52:59 on dist-test-slave-j2vl"
server_key: "ffd81a820370bbdbed39f1331f72e685"
server_key_iv: "794ce504bb354e70590d058dc421476b"
server_key_version: "encryptionkey@0"
I20260809 00:52:59.412571 19753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:52:59.428148 19753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:52:59.433782 19753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:52:59.434209 19753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:52:59.435446 19753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:52:59.437193 19753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:52:59.437282 19753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:59.437463 19753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:52:59.437523 19753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:52:59.477281 19753 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:40145
I20260809 00:52:59.477319 19988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:40145 every 8 connection(s)
I20260809 00:52:59.479053 19753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:52:59.480516 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 19753
I20260809 00:52:59.480737 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:52:59.481240 10116 external_mini_cluster.cc:1428] Setting key d5f230a8295a91f1c713db193558ccaf
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:52:59.506574 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:59.482935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:60829 (local address 127.9.225.62:42201)
0809 00:52:59.483528 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:59.483536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:59.485191 (+ 1655us) server_negotiation.cc:407] Connection header received
0809 00:52:59.487632 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:59.487638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:59.487720 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:59.487827 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:59.489763 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.490661 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.492476 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.492938 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.493056 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:59.498005 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:59.498014 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:59.498019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:59.498078 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:59.501873 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.503365 (+ 1492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.503375 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.503378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.503601 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.504290 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.504296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.504298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.504627 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:59.504867 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:59.505862 (+ 995us) server_negotiation.cc:299] Negotiation successful
0809 00:52:59.506158 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":286,"threads_started":1}
I20260809 00:52:59.507782 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:59.483963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:42201 (local address 127.9.225.3:60829)
0809 00:52:59.484977 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0809 00:52:59.485048 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0809 00:52:59.487283 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:52:59.488086 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:52:59.488096 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:52:59.488725 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:52:59.489520 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:59.489544 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.490887 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.490892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:59.492286 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:52:59.492303 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.493225 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.493232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:52:59.493601 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:59.494522 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:52:59.494550 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:52:59.497533 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:52:59.502214 (+ 4681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:59.502228 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:59.502256 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:59.503061 (+ 805us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:59.503746 (+ 685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:52:59.503750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:52:59.503753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:52:59.504124 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:52:59.504810 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:52:59.504821 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:52:59.505274 (+ 453us) client_negotiation.cc:769] Sending connection context
0809 00:52:59.505698 (+ 424us) client_negotiation.cc:240] Negotiation successful
0809 00:52:59.506389 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":430,"threads_started":1}
I20260809 00:52:59.509786 19989 heartbeater.cc:347] Connected to a master server at 127.9.225.62:42201
I20260809 00:52:59.510254 19989 heartbeater.cc:464] Registering TS with master...
I20260809 00:52:59.511361 19989 heartbeater.cc:511] Master 127.9.225.62:42201 requested a full tablet report, sending...
I20260809 00:52:59.513329 19194 ts_manager.cc:194] Registered new tserver with Master: f173bf2cb3994fbabd2f0ffae1af3ee7 (127.9.225.3:40145)
I20260809 00:52:59.514448 19194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:60829
I20260809 00:52:59.525962 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/test-admin.keytab.
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:59.565568 10116 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 00:52:59.572657 20006 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:52:59.569538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.9.225.1:38993)
0809 00:52:59.570286 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:59.570294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:59.570323 (+ 29us) server_negotiation.cc:407] Connection header received
0809 00:52:59.570399 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:59.570406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:59.570665 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:59.570863 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:59.571946 (+ 1083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49750: BlockingRecv error: recv got EOF from 127.0.0.1:49750 (error 108)
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":359,"threads_started":1}
W20260809 00:52:59.572710 10116 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/krb5cc' of type FILE
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:59.575248 10116 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:59.577250 10116 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:59.599753 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:59.580206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49752 (local address 127.9.225.1:38993)
0809 00:52:59.580548 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:59.580556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:59.580796 (+ 240us) server_negotiation.cc:407] Connection header received
0809 00:52:59.581073 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:59.581078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:59.581160 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:59.581282 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:59.582762 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.584078 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.585683 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.586438 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.586564 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:59.591074 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:59.591095 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:59.591116 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:59.591166 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:59.594822 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.596289 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.596303 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.596307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.596511 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.597456 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.597463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.597466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.598153 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:59.598532 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:59.599143 (+ 611us) server_negotiation.cc:299] Negotiation successful
0809 00:52:59.599442 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260809 00:52:59.607997 10116 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/krb5cc' of type FILE
W20260809 00:52:59.607975 20006 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:52:59.605105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49762 (local address 127.9.225.1:38993)
0809 00:52:59.605443 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:59.605450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:59.605701 (+ 251us) server_negotiation.cc:407] Connection header received
0809 00:52:59.605947 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:59.605952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:59.606032 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:59.606138 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:59.607598 (+ 1460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49762: BlockingRecv error: recv got EOF from 127.0.0.1:49762 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:59.609877 10116 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:59.611377 10116 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:59.628785 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:59.613201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.9.225.1:38993)
0809 00:52:59.613512 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:59.613519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:59.613790 (+ 271us) server_negotiation.cc:407] Connection header received
0809 00:52:59.614018 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:59.614022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:59.614096 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:59.614206 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:59.615301 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.616053 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.617516 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.617995 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.618143 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:59.622392 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:59.622399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:59.622402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:59.622448 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:59.625706 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.626634 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.626639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.626642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.626804 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.627262 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.627271 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.627273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.627578 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:59.627883 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:59.628210 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 00:52:59.628454 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260809 00:52:59.634292 20006 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:52:59.631968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.9.225.1:38993)
0809 00:52:59.632224 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:59.632234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:59.632548 (+ 314us) server_negotiation.cc:407] Connection header received
0809 00:52:59.632763 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:59.632767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:59.632848 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:59.632959 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:59.634046 (+ 1087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49780: BlockingRecv error: recv got EOF from 127.0.0.1:49780 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260809 00:52:59.634745 10116 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786236719337175-10116-0/krb5kdc/krb5cc' of type FILE
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:59.636557 10116 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:52:59.638147 10116 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:52:59.654438 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:59.640035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.9.225.1:38993)
0809 00:52:59.640261 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:59.640269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:59.640384 (+ 115us) server_negotiation.cc:407] Connection header received
0809 00:52:59.640638 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:59.640642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:59.640709 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:59.640820 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:59.641935 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.642710 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.644036 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.644480 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.644627 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:59.648832 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:59.648839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:59.648843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:59.648908 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:59.651747 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.652544 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.652548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.652551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.652711 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.653045 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.653049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.653051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.653334 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:59.653674 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:59.653939 (+ 265us) server_negotiation.cc:299] Negotiation successful
0809 00:52:59.654136 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 00:52:59.656631 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 19312
I20260809 00:52:59.670814 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:59.657485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.9.225.1:38993)
0809 00:52:59.657721 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:59.657725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:59.658108 (+ 383us) server_negotiation.cc:407] Connection header received
0809 00:52:59.658247 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:59.658251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:59.658316 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:59.658413 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:59.659513 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.660230 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.661742 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.662236 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.662358 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:59.664731 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:59.664736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:59.664740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:59.664779 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:59.667719 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.668688 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.668693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.668701 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.668857 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.669284 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.669288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.669291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.669741 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:59.669980 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:59.670295 (+ 315us) server_negotiation.cc:299] Negotiation successful
0809 00:52:59.670534 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 00:52:59.671546 19506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:59.762740 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 19312
I20260809 00:52:59.798049 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 19522
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:52:59.816213 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:59.799119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.9.225.2:40727)
0809 00:52:59.799766 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:59.799775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:59.799805 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:52:59.799873 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:59.799888 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:59.800121 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:59.800266 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:59.801498 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.802558 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.803965 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.804600 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.804713 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:59.809229 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:59.809248 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:59.809278 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:59.809397 (+ 119us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:59.812659 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.813688 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.813698 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.813701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.813885 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.814369 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.814373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.814376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.814853 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:59.815167 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:59.815518 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 00:52:59.815832 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":360,"threads_started":1}
I20260809 00:52:59.817766 19741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:52:59.906697 19747 heartbeater.cc:503] Master 127.9.225.62:42201 was elected leader, sending a full tablet report...
I20260809 00:52:59.907765 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 19522
W20260809 00:52:59.932478 19168 connection.cc:535] server connection from 127.9.225.2:46497 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 00:52:59.932652 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 19753
Aug 09 00:52:59 dist-test-slave-j2vl krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:52:59.951197 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:52:59.933828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.9.225.3:40145)
0809 00:52:59.934448 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0809 00:52:59.934456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:52:59.934490 (+ 34us) server_negotiation.cc:407] Connection header received
0809 00:52:59.934559 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:52:59.934566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:52:59.934825 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:52:59.935066 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:52:59.936213 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.937270 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.938656 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:52:59.939262 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:52:59.939427 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:52:59.944082 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:52:59.944102 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:52:59.944123 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:52:59.944172 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:52:59.947907 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.948889 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.948899 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.948901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.949078 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:52:59.949602 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:52:59.949607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:52:59.949609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:52:59.950025 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:52:59.950286 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0809 00:52:59.950610 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 00:52:59.950845 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":293,"threads_started":1}
I20260809 00:52:59.952703 19983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:00.041522 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 19753
I20260809 00:53:00.066525 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 19141
I20260809 00:53:00.067971 19253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:00.178009 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 19141
2026-08-09T00:53:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4850 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:53:00 dist-test-slave-j2vl krb5kdc[20040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:53:00 dist-test-slave-j2vl krb5kdc[20040](info): set up 2 sockets
Aug 09 00:53:00 dist-test-slave-j2vl krb5kdc[20040](info): commencing operation
krb5kdc: starting...
W20260809 00:53:01.571534 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.335s user 0.002s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:53:01 dist-test-slave-j2vl krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:53:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.62.keytab.
Entry for principal oryx/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.62.keytab.
I20260809 00:53:01.878356 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:33999
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.62.keytab
--principal=oryx/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:33999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:02.263574 20056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:02.264019 20056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:02.264199 20056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:02.271919 20056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:53:02.272127 20056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:02.272264 20056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:02.272373 20056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:53:02.272469 20056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:53:02.283993 20056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:33999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.62.keytab
--principal=oryx/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:33999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:02.285712 20056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:02.287442 20056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:02.306017 20064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:02.306126 20062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:02.306102 20066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:02.307093 20056 server_base.cc:1034] running on GCE node
I20260809 00:53:02.308096 20056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:02.309872 20056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:02.311120 20056 hybrid_clock.cc:648] HybridClock initialized: now 1786236782311069 us; error 39 us; skew 500 ppm
I20260809 00:53:02.311569 20056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:02 dist-test-slave-j2vl krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236782, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:02.315121 20056 init.cc:377] Logged in from keytab as oryx/127.9.225.62@KRBTEST.COM (short username oryx)
I20260809 00:53:02.316901 20056 webserver.cc:460] Webserver started at http://127.9.225.62:40683/ using document root <none> and password file <none>
I20260809 00:53:02.317780 20056 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:02.317901 20056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:02.318181 20056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:02.321641 20056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "4293b37228574799b2a8ee7508e605ef"
format_stamp: "Formatted at 2026-08-09 00:53:02 on dist-test-slave-j2vl"
server_key: "0348048ac7d7e0b943736608c909f519"
server_key_iv: "db31a6d4bd5f70fa0c71174df002d960"
server_key_version: "encryptionkey@0"
I20260809 00:53:02.322636 20056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "4293b37228574799b2a8ee7508e605ef"
format_stamp: "Formatted at 2026-08-09 00:53:02 on dist-test-slave-j2vl"
server_key: "0348048ac7d7e0b943736608c909f519"
server_key_iv: "db31a6d4bd5f70fa0c71174df002d960"
server_key_version: "encryptionkey@0"
I20260809 00:53:02.328605 20056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 00:53:02.332521 20078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:02.334178 20056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 00:53:02.334420 20056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "4293b37228574799b2a8ee7508e605ef"
format_stamp: "Formatted at 2026-08-09 00:53:02 on dist-test-slave-j2vl"
server_key: "0348048ac7d7e0b943736608c909f519"
server_key_iv: "db31a6d4bd5f70fa0c71174df002d960"
server_key_version: "encryptionkey@0"
I20260809 00:53:02.334630 20056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:02.358099 20056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:02.363976 20056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:02.364363 20056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:02.386574 20056 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:33999
I20260809 00:53:02.386605 20166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:33999 every 8 connection(s)
I20260809 00:53:02.388391 20056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:53:02.390487 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 20056
I20260809 00:53:02.390741 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:53:02.391629 10116 external_mini_cluster.cc:1428] Setting key 29622ea0edfdca9369594c22e323df33
I20260809 00:53:02.393841 20167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 00:53:02 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:02.407055 20167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef: Bootstrap starting.
I20260809 00:53:02.411053 20167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:02.412278 20167 log.cc:824] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:02.416330 20167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef: No bootstrap required, opened a new log
I20260809 00:53:02.417533 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:02.394139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.9.225.62:33999)
0809 00:53:02.395026 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:02.395039 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:02.395085 (+ 46us) server_negotiation.cc:407] Connection header received
0809 00:53:02.396886 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:02.396915 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:02.397469 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:02.397986 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:02.399672 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:02.401070 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:02.402554 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:02.403401 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:02.403664 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:02.408343 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:02.408353 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:02.408371 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:02.408421 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:02.411967 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:02.412944 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:02.412959 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:02.412969 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:02.413183 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:02.413732 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:02.413736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:02.413738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:02.414617 (+ 879us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:02.414993 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:02.415346 (+ 353us) server_negotiation.cc:299] Negotiation successful
0809 00:53:02.415848 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":377,"threads_started":1}
I20260809 00:53:02.425828 20167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } }
I20260809 00:53:02.426312 20167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:02.426395 20167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4293b37228574799b2a8ee7508e605ef, State: Initialized, Role: FOLLOWER
I20260809 00:53:02.427079 20167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } }
I20260809 00:53:02.427325 20167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:53:02.427439 20167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:53:02.427587 20167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:02.430150 20167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } }
I20260809 00:53:02.430729 20167 leader_election.cc:302] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4293b37228574799b2a8ee7508e605ef; no voters:
I20260809 00:53:02.431409 20167 leader_election.cc:288] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:53:02.431597 20175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:02.432996 20175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 1 LEADER]: Becoming Leader. State: Replica: 4293b37228574799b2a8ee7508e605ef, State: Running, Role: LEADER
I20260809 00:53:02.433809 20175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } }
I20260809 00:53:02.434170 20167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:53:02.437688 20177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4293b37228574799b2a8ee7508e605ef. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } }
I20260809 00:53:02.438122 20177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:02.438838 20176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } }
I20260809 00:53:02.439198 20176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:02.439229 20185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:53:02.446103 20185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:53:02.457216 20185 catalog_manager.cc:1347] Generated new cluster ID: 35e54945d8724482bf33ac4b98a68035
I20260809 00:53:02.457409 20185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:53:02.471153 20185 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:53:02.473658 20185 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:53:02.484076 20185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef: Generated new TSK 0
I20260809 00:53:02.485482 20185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab.
Entry for principal oryx/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab.
I20260809 00:53:02.563266 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33999
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab
--principal=oryx/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:02.966050 20203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:02.966398 20203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:02.966567 20203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:02.974485 20203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:02.974628 20203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:02.974850 20203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:53:02.985744 20203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab
--principal=oryx/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:02.987303 20203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:02.989007 20203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:03.004658 20211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:03.004554 20212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:03.004627 20214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:03.005803 20203 server_base.cc:1034] running on GCE node
I20260809 00:53:03.006583 20203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:03.007754 20203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:03.009035 20203 hybrid_clock.cc:648] HybridClock initialized: now 1786236783008984 us; error 45 us; skew 500 ppm
I20260809 00:53:03.009521 20203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:03 dist-test-slave-j2vl krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236783, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:03.024695 20203 init.cc:377] Logged in from keytab as oryx/127.9.225.1@KRBTEST.COM (short username oryx)
I20260809 00:53:03.026768 20203 webserver.cc:460] Webserver started at http://127.9.225.1:36525/ using document root <none> and password file <none>
I20260809 00:53:03.027626 20203 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:03.027803 20203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:03.028167 20203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:03.031649 20203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "14ac5161b2634e11a347324e9b1cb2e5"
format_stamp: "Formatted at 2026-08-09 00:53:03 on dist-test-slave-j2vl"
server_key: "710b28a571fe52fe20c42f6fe25eb740"
server_key_iv: "1ddea19d4880bdac8ac344c72353b10a"
server_key_version: "encryptionkey@0"
I20260809 00:53:03.032596 20203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "14ac5161b2634e11a347324e9b1cb2e5"
format_stamp: "Formatted at 2026-08-09 00:53:03 on dist-test-slave-j2vl"
server_key: "710b28a571fe52fe20c42f6fe25eb740"
server_key_iv: "1ddea19d4880bdac8ac344c72353b10a"
server_key_version: "encryptionkey@0"
I20260809 00:53:03.039217 20203 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260809 00:53:03.043506 20224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:03.045087 20203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 00:53:03.045316 20203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "14ac5161b2634e11a347324e9b1cb2e5"
format_stamp: "Formatted at 2026-08-09 00:53:03 on dist-test-slave-j2vl"
server_key: "710b28a571fe52fe20c42f6fe25eb740"
server_key_iv: "1ddea19d4880bdac8ac344c72353b10a"
server_key_version: "encryptionkey@0"
I20260809 00:53:03.045626 20203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:03.072506 20203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:03.078629 20203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:03.079084 20203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:03.080564 20203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:03.082738 20203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:03.082878 20203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:03.083077 20203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:03.083158 20203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:03.120801 20203 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:46803
I20260809 00:53:03.120850 20418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:46803 every 8 connection(s)
I20260809 00:53:03.122298 20203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:53:03.129859 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 20203
I20260809 00:53:03.130118 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:53:03.131021 10116 external_mini_cluster.cc:1428] Setting key 5b21028f5bd478d40aee0545c8749d6a
Aug 09 00:53:03 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236783, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:03.148263 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:03.125446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:46825 (local address 127.9.225.62:33999)
0809 00:53:03.125992 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:03.125999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:03.127153 (+ 1154us) server_negotiation.cc:407] Connection header received
0809 00:53:03.129087 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:03.129093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:03.129171 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:03.129290 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:03.131813 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:03.132881 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:03.135709 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:03.136165 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:03.136295 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:03.141629 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:03.141635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:03.141639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:03.141680 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:03.144563 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:03.145535 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:03.145548 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:03.145551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:03.145697 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:03.146279 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:03.146283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:03.146285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:03.146594 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:03.146715 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:03.147765 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0809 00:53:03.147975 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":241,"threads_started":1}
I20260809 00:53:03.150142 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:03.126121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.1:46825)
0809 00:53:03.126967 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:03.127014 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:03.128760 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:03.129650 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:03.129664 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:03.130562 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:03.131520 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:03.131550 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:03.133104 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:03.133113 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:03.135425 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:03.135451 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:03.136445 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:03.136457 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:03.136834 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:03.137733 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:03.137773 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:03.141270 (+ 3497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:03.144756 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:03.144777 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:03.144802 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:03.145303 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:03.145835 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:03.145844 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:03.145850 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:03.146152 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:03.146743 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:03.146755 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:03.147298 (+ 543us) client_negotiation.cc:769] Sending connection context
0809 00:53:03.147648 (+ 350us) client_negotiation.cc:240] Negotiation successful
0809 00:53:03.148386 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":327,"threads_started":1}
I20260809 00:53:03.152074 20419 heartbeater.cc:347] Connected to a master server at 127.9.225.62:33999
I20260809 00:53:03.152467 20419 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:03.153324 20419 heartbeater.cc:511] Master 127.9.225.62:33999 requested a full tablet report, sending...
I20260809 00:53:03.156267 20107 ts_manager.cc:194] Registered new tserver with Master: 14ac5161b2634e11a347324e9b1cb2e5 (127.9.225.1:46803)
I20260809 00:53:03.158104 20107 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:46825
WARNING: no policy specified for oryx/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab.
Entry for principal oryx/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab.
I20260809 00:53:03.201928 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33999
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab
--principal=oryx/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:03.576126 20428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:03.576526 20428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:03.576712 20428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:03.584635 20428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:03.584846 20428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:03.585037 20428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:53:03.596719 20428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab
--principal=oryx/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:03.598459 20428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:03.600294 20428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:03.616075 20436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:03.616110 20439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:03.616153 20443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:03.617204 20428 server_base.cc:1034] running on GCE node
I20260809 00:53:03.617981 20428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:03.619099 20428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:03.620358 20428 hybrid_clock.cc:648] HybridClock initialized: now 1786236783620313 us; error 50 us; skew 500 ppm
I20260809 00:53:03.620733 20428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:03 dist-test-slave-j2vl krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236783, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:03.624414 20428 init.cc:377] Logged in from keytab as oryx/127.9.225.2@KRBTEST.COM (short username oryx)
I20260809 00:53:03.626471 20428 webserver.cc:460] Webserver started at http://127.9.225.2:35055/ using document root <none> and password file <none>
I20260809 00:53:03.627270 20428 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:03.627388 20428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:03.627679 20428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:03.631328 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "d4de19738ca547b38df7a46b3eae96a6"
format_stamp: "Formatted at 2026-08-09 00:53:03 on dist-test-slave-j2vl"
server_key: "59334a7203908e313e82b1f6933e19b5"
server_key_iv: "7acc46e7aab26ce47e747dd1ca7b12e1"
server_key_version: "encryptionkey@0"
I20260809 00:53:03.632239 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "d4de19738ca547b38df7a46b3eae96a6"
format_stamp: "Formatted at 2026-08-09 00:53:03 on dist-test-slave-j2vl"
server_key: "59334a7203908e313e82b1f6933e19b5"
server_key_iv: "7acc46e7aab26ce47e747dd1ca7b12e1"
server_key_version: "encryptionkey@0"
I20260809 00:53:03.639101 20428 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 00:53:03.643410 20454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:03.645159 20428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 00:53:03.645429 20428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "d4de19738ca547b38df7a46b3eae96a6"
format_stamp: "Formatted at 2026-08-09 00:53:03 on dist-test-slave-j2vl"
server_key: "59334a7203908e313e82b1f6933e19b5"
server_key_iv: "7acc46e7aab26ce47e747dd1ca7b12e1"
server_key_version: "encryptionkey@0"
I20260809 00:53:03.645681 20428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:03.663060 20428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:03.669838 20428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:03.670295 20428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:03.671876 20428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:03.678189 20428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:03.678367 20428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:03.678524 20428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:03.678606 20428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:03.727751 20428 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:37159
I20260809 00:53:03.727809 20623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:37159 every 8 connection(s)
I20260809 00:53:03.729652 20428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:53:03.736984 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 20428
I20260809 00:53:03.737202 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:53:03.737816 10116 external_mini_cluster.cc:1428] Setting key 7319605829baa41b14a89bdcb914339f
Aug 09 00:53:03 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236783, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:03.759662 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:03.734707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:48131 (local address 127.9.225.62:33999)
0809 00:53:03.735301 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:03.735310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:03.735963 (+ 653us) server_negotiation.cc:407] Connection header received
0809 00:53:03.739033 (+ 3070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:03.739041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:03.739200 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:03.739311 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:03.742129 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:03.743251 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:03.745909 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:03.746641 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:03.746804 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:03.751898 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:03.751909 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:03.751916 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:03.751975 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:03.755708 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:03.756719 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:03.756725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:03.756729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:03.756891 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:03.757429 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:03.757435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:03.757438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:03.757786 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:03.757971 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:03.758966 (+ 995us) server_negotiation.cc:299] Negotiation successful
0809 00:53:03.759249 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":256,"threads_started":1}
I20260809 00:53:03.760887 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:03.733980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.2:48131)
0809 00:53:03.735734 (+ 1754us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:03.735802 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:03.738506 (+ 2704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:03.739800 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:03.739815 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:03.740795 (+ 980us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:03.741873 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:03.741900 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:03.743465 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:03.743470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:03.745724 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:03.745747 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:03.746927 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:03.746936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:03.747225 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:03.748035 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:03.748062 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:03.751527 (+ 3465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:03.755970 (+ 4443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:03.755982 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:03.756016 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:03.756557 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:03.757020 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:03.757025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:03.757029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:03.757263 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:03.757942 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:03.757951 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:03.758377 (+ 426us) client_negotiation.cc:769] Sending connection context
0809 00:53:03.758841 (+ 464us) client_negotiation.cc:240] Negotiation successful
0809 00:53:03.759530 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1365,"thread_start_us":1026,"threads_started":1}
I20260809 00:53:03.762796 20625 heartbeater.cc:347] Connected to a master server at 127.9.225.62:33999
I20260809 00:53:03.763458 20625 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:03.764694 20625 heartbeater.cc:511] Master 127.9.225.62:33999 requested a full tablet report, sending...
I20260809 00:53:03.767455 20107 ts_manager.cc:194] Registered new tserver with Master: d4de19738ca547b38df7a46b3eae96a6 (127.9.225.2:37159)
I20260809 00:53:03.768440 20107 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:48131
WARNING: no policy specified for oryx/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab.
Entry for principal oryx/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab.
I20260809 00:53:03.808651 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33999
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab
--principal=oryx/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:53:04.161989 20419 heartbeater.cc:503] Master 127.9.225.62:33999 was elected leader, sending a full tablet report...
W20260809 00:53:04.188231 20635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:04.188591 20635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:04.188692 20635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:04.196759 20635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:04.196924 20635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:04.197047 20635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:53:04.207726 20635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab
--principal=oryx/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:33999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:04.209192 20635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:04.210752 20635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:04.225925 20644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:04.225840 20645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:04.225908 20647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:04.226984 20635 server_base.cc:1034] running on GCE node
I20260809 00:53:04.227866 20635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:04.229013 20635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:04.230252 20635 hybrid_clock.cc:648] HybridClock initialized: now 1786236784230167 us; error 72 us; skew 500 ppm
I20260809 00:53:04.230608 20635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:04 dist-test-slave-j2vl krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236784, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:04.234396 20635 init.cc:377] Logged in from keytab as oryx/127.9.225.3@KRBTEST.COM (short username oryx)
I20260809 00:53:04.236335 20635 webserver.cc:460] Webserver started at http://127.9.225.3:40767/ using document root <none> and password file <none>
I20260809 00:53:04.237170 20635 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:04.237336 20635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:04.237802 20635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:04.241518 20635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "0dbc6b27c27b46b094da602fcd3d99c8"
format_stamp: "Formatted at 2026-08-09 00:53:04 on dist-test-slave-j2vl"
server_key: "9ef939bc34bdf600eb11c8e2a25e9213"
server_key_iv: "7e3c256e416208852c15860f5ed5c4c3"
server_key_version: "encryptionkey@0"
I20260809 00:53:04.242408 20635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "0dbc6b27c27b46b094da602fcd3d99c8"
format_stamp: "Formatted at 2026-08-09 00:53:04 on dist-test-slave-j2vl"
server_key: "9ef939bc34bdf600eb11c8e2a25e9213"
server_key_iv: "7e3c256e416208852c15860f5ed5c4c3"
server_key_version: "encryptionkey@0"
I20260809 00:53:04.248957 20635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 00:53:04.253520 20661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:04.255249 20635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260809 00:53:04.255429 20635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "0dbc6b27c27b46b094da602fcd3d99c8"
format_stamp: "Formatted at 2026-08-09 00:53:04 on dist-test-slave-j2vl"
server_key: "9ef939bc34bdf600eb11c8e2a25e9213"
server_key_iv: "7e3c256e416208852c15860f5ed5c4c3"
server_key_version: "encryptionkey@0"
I20260809 00:53:04.255636 20635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:04.269750 20635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:04.274704 20635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:04.275043 20635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:04.276093 20635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:04.277689 20635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:04.277769 20635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:04.277913 20635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:04.277967 20635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:04.317286 20635 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:45797
I20260809 00:53:04.317322 20839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:45797 every 8 connection(s)
I20260809 00:53:04.318852 20635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:53:04.329324 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 20635
I20260809 00:53:04.329596 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:53:04.330106 10116 external_mini_cluster.cc:1428] Setting key b4d313961e97dc2ac13be2c88874b839
Aug 09 00:53:04 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236784, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:04.344314 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:04.322133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:40317 (local address 127.9.225.62:33999)
0809 00:53:04.322668 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:04.322673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:04.324389 (+ 1716us) server_negotiation.cc:407] Connection header received
0809 00:53:04.326989 (+ 2600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:04.326996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:04.327107 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:04.327243 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:04.329161 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.330174 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.332322 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.332950 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.333098 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:04.337458 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:04.337465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:04.337468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:04.337509 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:04.340784 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:04.341818 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:04.341823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:04.341825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:04.341970 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:04.342462 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:04.342466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:04.342469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:04.342812 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:04.342955 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:04.343757 (+ 802us) server_negotiation.cc:299] Negotiation successful
0809 00:53:04.343988 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":273,"threads_started":1}
I20260809 00:53:04.345696 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:04.322792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.3:40317)
0809 00:53:04.324057 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:04.324144 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:04.326540 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:04.327551 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:04.327561 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:04.328196 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:04.328925 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:04.328947 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.330432 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.330440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:04.332122 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:04.332152 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.333203 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.333209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:04.333549 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:04.334426 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:04.334454 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:04.337104 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:04.340993 (+ 3889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:04.341002 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:04.341020 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:04.341639 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:04.342083 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:04.342087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:04.342089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:04.342305 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:04.342950 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:04.342957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:04.343330 (+ 373us) client_negotiation.cc:769] Sending connection context
0809 00:53:04.343612 (+ 282us) client_negotiation.cc:240] Negotiation successful
0809 00:53:04.344082 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":461,"threads_started":1}
I20260809 00:53:04.347487 20840 heartbeater.cc:347] Connected to a master server at 127.9.225.62:33999
I20260809 00:53:04.347885 20840 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:04.348749 20840 heartbeater.cc:511] Master 127.9.225.62:33999 requested a full tablet report, sending...
I20260809 00:53:04.350898 20106 ts_manager.cc:194] Registered new tserver with Master: 0dbc6b27c27b46b094da602fcd3d99c8 (127.9.225.3:45797)
I20260809 00:53:04.352041 20106 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:40317
I20260809 00:53:04.360673 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.225.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.61.keytab.
Entry for principal oryx/127.9.225.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.61.keytab.
I20260809 00:53:04.429610 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.61:37619
--webserver_interface=127.9.225.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.61.keytab
--principal=oryx/127.9.225.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:33999,127.9.225.61:37619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:53:04.771816 20625 heartbeater.cc:503] Master 127.9.225.62:33999 was elected leader, sending a full tablet report...
W20260809 00:53:04.820446 20847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:04.820894 20847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:04.821007 20847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:04.828660 20847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:53:04.828789 20847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:04.828837 20847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:04.828895 20847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:53:04.828933 20847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:53:04.839787 20847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:33999,127.9.225.61:37619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.61.keytab
--principal=oryx/127.9.225.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.61:37619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.225.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
W20260809 00:53:04.840214 20847 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.225.62:33999,127.9.225.61:37619'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 00:53:04.841331 20847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:04.842809 20847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:04.855540 20854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:04.855623 20859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:04.855654 20856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:04.856673 20847 server_base.cc:1034] running on GCE node
I20260809 00:53:04.857625 20847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:04.859270 20847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:04.860586 20847 hybrid_clock.cc:648] HybridClock initialized: now 1786236784860483 us; error 67 us; skew 500 ppm
I20260809 00:53:04.861025 20847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:04 dist-test-slave-j2vl krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236784, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:04.864418 20847 init.cc:377] Logged in from keytab as oryx/127.9.225.61@KRBTEST.COM (short username oryx)
I20260809 00:53:04.866084 20847 webserver.cc:460] Webserver started at http://127.9.225.61:46549/ using document root <none> and password file <none>
I20260809 00:53:04.866796 20847 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:04.866885 20847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:04.867157 20847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:04.870426 20847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data/instance:
uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba"
format_stamp: "Formatted at 2026-08-09 00:53:04 on dist-test-slave-j2vl"
server_key: "eed6f832c856fe38392f854f76832d7b"
server_key_iv: "71cb24f3336a73e5d5f3d3b0a21e81c0"
server_key_version: "encryptionkey@0"
I20260809 00:53:04.871210 20847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/wal/instance:
uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba"
format_stamp: "Formatted at 2026-08-09 00:53:04 on dist-test-slave-j2vl"
server_key: "eed6f832c856fe38392f854f76832d7b"
server_key_iv: "71cb24f3336a73e5d5f3d3b0a21e81c0"
server_key_version: "encryptionkey@0"
I20260809 00:53:04.876829 20847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 00:53:04.881796 20873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:04.883365 20847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 00:53:04.883538 20847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/wal
uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba"
format_stamp: "Formatted at 2026-08-09 00:53:04 on dist-test-slave-j2vl"
server_key: "eed6f832c856fe38392f854f76832d7b"
server_key_iv: "71cb24f3336a73e5d5f3d3b0a21e81c0"
server_key_version: "encryptionkey@0"
I20260809 00:53:04.883746 20847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:04.903553 20847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:04.910989 20847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:04.911340 20847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:04.939097 20847 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.61:37619
I20260809 00:53:04.939141 20955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.61:37619 every 8 connection(s)
I20260809 00:53:04.941862 20847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
I20260809 00:53:04.949079 20957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:04.951409 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 20847
I20260809 00:53:04.965658 20957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:53:04 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.225.61@KRBTEST.COM
Aug 09 00:53:04 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236784, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.61@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:04.986810 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:04.955447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.9.225.61:37619)
0809 00:53:04.956885 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:04.956909 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:04.956998 (+ 89us) server_negotiation.cc:407] Connection header received
0809 00:53:04.959516 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:04.959580 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:04.960664 (+ 1084us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:04.961484 (+ 820us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:04.964486 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.966720 (+ 2234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.969691 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.971409 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.971823 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:04.978397 (+ 6574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:04.978406 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:04.978420 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:04.978471 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:04.981980 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:04.982897 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:04.982906 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:04.982915 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:04.983124 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:04.983665 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:04.983670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:04.983672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:04.984407 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:04.984661 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:04.985228 (+ 567us) server_negotiation.cc:299] Negotiation successful
0809 00:53:04.985732 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":613,"threads_started":1}
I20260809 00:53:04.990223 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:04.971970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.9.225.62:33999)
0809 00:53:04.972782 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:04.972790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:04.973733 (+ 943us) server_negotiation.cc:407] Connection header received
0809 00:53:04.974018 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:04.974023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:04.974098 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:04.974185 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:04.976006 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.977186 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.979189 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.979678 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.979851 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:04.983535 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:04.983545 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:04.983549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:04.983611 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:04.986804 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:04.987733 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:04.987737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:04.987740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:04.987883 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:04.988356 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:04.988366 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:04.988368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:04.988763 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:04.988962 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:04.989562 (+ 600us) server_negotiation.cc:299] Negotiation successful
0809 00:53:04.989813 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":348,"threads_started":1}
I20260809 00:53:04.990274 20965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:04.971915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:55898)
0809 00:53:04.973512 (+ 1597us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:04.973592 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:04.973827 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:04.974321 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:04.974332 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:04.975387 (+ 1055us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:04.975816 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:04.975832 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.977397 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.977404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:04.978928 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:04.978946 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.979897 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.979905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:04.980106 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:04.980723 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:04.980741 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:04.983321 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:04.986962 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:04.986971 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:04.986978 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:04.987603 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:04.988021 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:04.988025 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:04.988028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:04.988207 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:04.988912 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:04.988920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:04.989146 (+ 226us) client_negotiation.cc:769] Sending connection context
0809 00:53:04.989441 (+ 295us) client_negotiation.cc:240] Negotiation successful
0809 00:53:04.989811 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"thread_start_us":790,"threads_started":1}
I20260809 00:53:04.992463 20957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:53:04 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236784, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.61@KRBTEST.COM for oryx/127.9.225.61@KRBTEST.COM
I20260809 00:53:05.005533 20965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:04.993403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:37619 (local address 127.0.0.1:48806)
0809 00:53:04.993621 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:04.993642 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:04.993787 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:04.994131 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:04.994135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:04.994516 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:04.994662 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:04.994670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.995662 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.995666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:04.996575 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:04.996586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.997269 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.997273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:04.997520 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:04.998109 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:04.998127 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:05.000231 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:05.003320 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:05.003323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:05.003326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:05.003719 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:05.004114 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:05.004118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:05.004120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:05.004245 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:05.004711 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:05.004714 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:05.004847 (+ 133us) client_negotiation.cc:769] Sending connection context
0809 00:53:05.004964 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 00:53:05.005201 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 00:53:05.005738 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:04.993279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.9.225.61:37619)
0809 00:53:04.993480 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:04.993492 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:04.993710 (+ 218us) server_negotiation.cc:407] Connection header received
0809 00:53:04.993921 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:04.993925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:04.993984 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:04.994064 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:04.994801 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.995524 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.996709 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:04.997119 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:04.997216 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:05.000418 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:05.000424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:05.000427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:05.000457 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:05.003179 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:05.003832 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:05.003835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:05.003838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:05.004003 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:05.004334 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:05.004337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:05.004339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:05.004606 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:05.004719 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:05.005060 (+ 341us) server_negotiation.cc:299] Negotiation successful
0809 00:53:05.005405 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":46}
I20260809 00:53:05.010835 20957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: Bootstrap starting.
I20260809 00:53:05.014477 20957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:05.015913 20957 log.cc:824] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:05.018703 20957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: No bootstrap required, opened a new log
I20260809 00:53:05.026021 20957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } }
I20260809 00:53:05.026417 20957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:05.026499 20957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e34bc5f7a1944d0b6742a6ad64df4ba, State: Initialized, Role: FOLLOWER
I20260809 00:53:05.027184 20957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } }
I20260809 00:53:05.028129 20968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } } }
I20260809 00:53:05.028535 20968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:53:05.029327 20957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: configured and running, proceeding with master startup.
W20260809 00:53:05.040040 20986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 00:53:05.040227 20986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 00:53:05.044476 20847 master_runner.cc:428] Detected that this master 4e34bc5f7a1944d0b6742a6ad64df4ba is joining an existing cluster
I20260809 00:53:05.044620 20847 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.225.61:37619
I20260809 00:53:05.077750 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:05.054692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.9.225.62:33999)
0809 00:53:05.054898 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:05.054905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:05.056591 (+ 1686us) server_negotiation.cc:407] Connection header received
0809 00:53:05.056851 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:05.056860 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:05.056967 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:05.057071 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:05.058973 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.060313 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.063160 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.063966 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.064117 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:05.068090 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:05.068100 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:05.068108 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:05.068164 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:05.072573 (+ 4409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:05.074260 (+ 1687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:05.074267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:05.074270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:05.074458 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:05.075032 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:05.075037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:05.075040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:05.075468 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:05.075619 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:05.076676 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0809 00:53:05.076994 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
WARNING: no policy specified for oryx/127.9.225.60@KRBTEST.COM; defaulting to no policy
I20260809 00:53:05.079260 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:05.054036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:55914)
0809 00:53:05.056420 (+ 2384us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:05.056448 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:05.056686 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:05.057303 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:05.057311 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:05.058145 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:05.058670 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:05.058691 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.060787 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.060795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:05.062923 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:05.062949 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.064557 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.064566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:05.064923 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:05.066105 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:05.066137 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:05.067767 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:05.072933 (+ 5166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:05.072940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:05.072946 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:05.074065 (+ 1119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:05.074642 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:05.074660 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:05.074669 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:05.074886 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:05.075716 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:05.075723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:05.076324 (+ 601us) client_negotiation.cc:769] Sending connection context
0809 00:53:05.078083 (+ 1759us) client_negotiation.cc:240] Negotiation successful
0809 00:53:05.078550 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2225,"thread_start_us":11632,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.60@KRBTEST.COM" created.
I20260809 00:53:05.084796 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:05.054061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.9.225.61:37619)
0809 00:53:05.054284 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:05.054293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:05.066034 (+ 11741us) server_negotiation.cc:407] Connection header received
0809 00:53:05.066282 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:05.066288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:05.066359 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:05.066433 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:05.068015 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.069194 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.071282 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.072011 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.072126 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:05.075011 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:05.075018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:05.075022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:05.075055 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:05.079270 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:05.081480 (+ 2210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:05.081486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:05.081490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:05.081654 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:05.082199 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:05.082210 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:05.082216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:05.082647 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:05.082823 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:05.083753 (+ 930us) server_negotiation.cc:299] Negotiation successful
0809 00:53:05.084270 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 00:53:05.086094 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:05.054548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:37619 (local address 127.0.0.1:48820)
0809 00:53:05.065928 (+ 11380us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:05.065950 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:05.066139 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:05.066658 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:05.066663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:05.067449 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:05.067848 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:05.067862 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.069467 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.069473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:05.071106 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:05.071129 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.072479 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.072485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:05.072672 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:05.073422 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:05.073447 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:05.074780 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:05.079475 (+ 4695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:05.079484 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:05.079490 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:05.081288 (+ 1798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:05.081862 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:05.081867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:05.081871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:05.082080 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:05.082912 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:05.082917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:05.083099 (+ 182us) client_negotiation.cc:769] Sending connection context
0809 00:53:05.085093 (+ 1994us) client_negotiation.cc:240] Negotiation successful
0809 00:53:05.085507 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11157,"thread_start_us":1541,"threads_started":1}
Aug 09 00:53:05 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236782, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.62@KRBTEST.COM for oryx/127.9.225.61@KRBTEST.COM
I20260809 00:53:05.110556 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:05.088008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:37619 (local address 127.0.0.1:48826)
0809 00:53:05.089002 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:05.089063 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:05.089404 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:05.090150 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:05.090162 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:05.091256 (+ 1094us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:05.091637 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:05.091656 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.093258 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.093267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:05.095303 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:05.095330 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.096539 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.096549 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:05.096837 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:05.097922 (+ 1085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:05.097961 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:05.101547 (+ 3586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:05.106385 (+ 4838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:05.106410 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:05.106421 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:05.107215 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:05.107891 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:05.107898 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:05.107904 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:05.108146 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:05.109071 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:05.109082 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:05.109314 (+ 232us) client_negotiation.cc:769] Sending connection context
0809 00:53:05.109614 (+ 300us) client_negotiation.cc:240] Negotiation successful
0809 00:53:05.110047 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":518,"threads_started":1}
I20260809 00:53:05.110584 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:05.088369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.9.225.61:37619)
0809 00:53:05.088572 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:05.088580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:05.089152 (+ 572us) server_negotiation.cc:407] Connection header received
0809 00:53:05.089705 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:05.089713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:05.089837 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:05.089963 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:05.091860 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.093072 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.095531 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.096293 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.096478 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:05.101901 (+ 5423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:05.101910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:05.101915 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:05.101968 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:05.106113 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:05.107460 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:05.107471 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:05.107477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:05.107710 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:05.108325 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:05.108333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:05.108337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:05.108890 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:05.109075 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:05.109790 (+ 715us) server_negotiation.cc:299] Negotiation successful
0809 00:53:05.110105 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 00:53:05.113595 20106 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.225.61:37619: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba"
member_type: NON_VOTER
last_known_addr {
host: "127.9.225.61"
port: 37619
}
attrs {
promote: true
}
}
dest_uuid: "4293b37228574799b2a8ee7508e605ef"
cas_config_opid_index: -1
I20260809 00:53:05.115096 20106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } }
I20260809 00:53:05.119019 21003 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4e34bc5f7a1944d0b6742a6ad64df4ba (127.9.225.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } } }
I20260809 00:53:05.120484 20912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.60.keytab.
Entry for principal oryx/127.9.225.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.60.keytab.
I20260809 00:53:05.124047 21003 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.225.61:37619
I20260809 00:53:05.126003 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.60:36845
--webserver_interface=127.9.225.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.60.keytab
--principal=oryx/127.9.225.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:33999,127.9.225.61:37619,127.9.225.60:36845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:53:05.126317 21003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } } }
I20260809 00:53:05.126595 21003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:05.129602 20912 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 FOLLOWER]: Refusing update from remote peer 4293b37228574799b2a8ee7508e605ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 00:53:05.130378 20847 master.cc:537] Master@127.9.225.61:37619 shutting down...
I20260809 00:53:05.131379 21002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 00:53:05.142577 20082 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260809 00:53:05.144896 20082 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef -> Peer 4e34bc5f7a1944d0b6742a6ad64df4ba (127.9.225.61:37619): Couldn't send request to peer 4e34bc5f7a1944d0b6742a6ad64df4ba. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 00:53:05.149724 20847 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:05.150352 20847 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:05.150573 20847 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: stopping tablet replica
I20260809 00:53:05.158145 20847 master.cc:552] Master@127.9.225.61:37619 shutdown complete.
I20260809 00:53:05.158320 20847 master_runner.cc:305] Clearing existing system tablet
I20260809 00:53:05.158849 20847 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:05.159073 20847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:05.162482 20847 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.005s sys 0.000s
I20260809 00:53:05.164542 21011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:05.165495 20847 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:05.165802 20847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 00:53:05.165895 20847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/wal
uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba"
format_stamp: "Formatted at 2026-08-09 00:53:04 on dist-test-slave-j2vl"
server_key: "eed6f832c856fe38392f854f76832d7b"
server_key_iv: "71cb24f3336a73e5d5f3d3b0a21e81c0"
server_key_version: "encryptionkey@0"
I20260809 00:53:05.171991 20847 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:05.180258 20847 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 00:53:05.180572 20847 log.cc:1196] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 00:53:05.181143 20847 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: Deleting consensus metadata
I20260809 00:53:05.181514 20847 master_runner.cc:315] Copying system tablet from 127.9.225.62:33999
I20260809 00:53:05.187505 20847 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: tablet copy: Beginning tablet copy session from remote peer at address 127.9.225.62:33999
I20260809 00:53:05.205924 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:05.188751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.9.225.62:33999)
0809 00:53:05.188985 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:05.188993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:05.189454 (+ 461us) server_negotiation.cc:407] Connection header received
0809 00:53:05.189701 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:05.189708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:05.189801 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:05.189921 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:05.191134 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.192509 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.194650 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.195427 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.195579 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:05.198368 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:05.198375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:05.198380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:05.198431 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:05.202243 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:05.203235 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:05.203242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:05.203246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:05.203420 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:05.203992 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:05.203998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:05.204001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:05.204411 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:05.204577 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:05.205185 (+ 608us) server_negotiation.cc:299] Negotiation successful
0809 00:53:05.205490 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 00:53:05.206579 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:05.188722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:55926)
0809 00:53:05.189286 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:05.189319 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:05.189532 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:05.189990 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:05.189996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:05.190599 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:05.190909 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:05.190934 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.192762 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.192768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:05.194396 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:05.194431 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.195657 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.195666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:05.195906 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:05.196778 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:05.196810 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:05.198077 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:05.202461 (+ 4384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:05.202467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:05.202472 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:05.203060 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:05.203579 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:05.203588 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:05.203595 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:05.203834 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:05.204579 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:05.204586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:05.204840 (+ 254us) client_negotiation.cc:769] Sending connection context
0809 00:53:05.205038 (+ 198us) client_negotiation.cc:240] Negotiation successful
0809 00:53:05.205550 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":304,"threads_started":1}
I20260809 00:53:05.207648 20139 tablet_copy_service.cc:140] P 4293b37228574799b2a8ee7508e605ef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4e34bc5f7a1944d0b6742a6ad64df4ba ({username='oryx', principal='oryx/127.9.225.61@KRBTEST.COM'} at 127.0.0.1:55926)
I20260809 00:53:05.208073 20139 tablet_copy_service.cc:161] P 4293b37228574799b2a8ee7508e605ef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4e34bc5f7a1944d0b6742a6ad64df4ba at {username='oryx', principal='oryx/127.9.225.61@KRBTEST.COM'} at 127.0.0.1:55926: session id = 4e34bc5f7a1944d0b6742a6ad64df4ba-00000000000000000000000000000000
I20260809 00:53:05.211676 20139 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 00:53:05.214695 20847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 00:53:05.222437 20847 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: tablet copy: Starting download of 0 data blocks...
I20260809 00:53:05.222916 20847 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: tablet copy: Starting download of 1 WAL segments...
I20260809 00:53:05.227368 20847 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 00:53:05.233024 20139 tablet_copy_service.cc:342] P 4293b37228574799b2a8ee7508e605ef: Request end of tablet copy session 4e34bc5f7a1944d0b6742a6ad64df4ba-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.225.61@KRBTEST.COM'} at 127.0.0.1:55926
I20260809 00:53:05.233408 20139 tablet_copy_service.cc:433] P 4293b37228574799b2a8ee7508e605ef: ending tablet copy session 4e34bc5f7a1944d0b6742a6ad64df4ba-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4e34bc5f7a1944d0b6742a6ad64df4ba
I20260809 00:53:05.247740 20847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:05.251551 21020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:05.255119 21022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:05.266702 20847 server_base.cc:1034] running on GCE node
W20260809 00:53:05.267087 21025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:05.267652 20847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:05.268432 20847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:05.269758 20847 hybrid_clock.cc:648] HybridClock initialized: now 1786236785269700 us; error 35 us; skew 500 ppm
I20260809 00:53:05.270067 20847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:05 dist-test-slave-j2vl krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236785, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:05.272128 20847 init.cc:377] Logged in from keytab as oryx/127.9.225.61@KRBTEST.COM (short username oryx)
I20260809 00:53:05.279713 20847 webserver.cc:460] Webserver started at http://127.9.225.61:33181/ using document root <none> and password file <none>
I20260809 00:53:05.280135 20847 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:05.280251 20847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:05.283039 20847 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260809 00:53:05.285051 21037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:05.285591 20847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260809 00:53:05.285760 20847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/wal
uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba"
format_stamp: "Formatted at 2026-08-09 00:53:04 on dist-test-slave-j2vl"
server_key: "eed6f832c856fe38392f854f76832d7b"
server_key_iv: "71cb24f3336a73e5d5f3d3b0a21e81c0"
server_key_version: "encryptionkey@0"
I20260809 00:53:05.285931 20847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:05.297650 20847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:05.301579 20847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:05.329217 20847 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.61:37619
I20260809 00:53:05.329246 21116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.61:37619 every 8 connection(s)
I20260809 00:53:05.330390 20847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
I20260809 00:53:05.336421 21117 sys_catalog.cc:263] Verifying existing consensus state
I20260809 00:53:05.340831 21117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: Bootstrap starting.
I20260809 00:53:05.355415 20840 heartbeater.cc:503] Master 127.9.225.62:33999 was elected leader, sending a full tablet report...
I20260809 00:53:05.363310 21117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 00:53:05.363969 21117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: Bootstrap complete.
I20260809 00:53:05.365181 21117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } }
I20260809 00:53:05.365502 21117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e34bc5f7a1944d0b6742a6ad64df4ba, State: Initialized, Role: LEARNER
I20260809 00:53:05.365789 21117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } }
I20260809 00:53:05.366698 21121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } } }
I20260809 00:53:05.367012 21117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:53:05.367189 21121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: This master's current role is: LEARNER
I20260809 00:53:05.375739 21141 catalog_manager.cc:1259] Loaded cluster ID: 35e54945d8724482bf33ac4b98a68035
I20260809 00:53:05.375928 21141 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: loading cluster ID for follower catalog manager: success
I20260809 00:53:05.379544 21141 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: acquiring CA information for follower catalog manager: success
I20260809 00:53:05.382550 21141 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 00:53:05.583835 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:05.569033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:37619 (local address 127.0.0.1:54710)
0809 00:53:05.569550 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:05.569583 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:05.569804 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:05.571389 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:05.571395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:05.571450 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:05.571752 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:05.571768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.575294 (+ 3526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.575301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:05.578765 (+ 3464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:05.578794 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.582488 (+ 3694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.582497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:05.582927 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:05.583032 (+ 105us) client_negotiation.cc:769] Sending connection context
0809 00:53:05.583160 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 00:53:05.583348 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260809 00:53:05.584532 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:05.569322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.9.225.61:37619)
0809 00:53:05.570814 (+ 1492us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:05.570822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:05.570857 (+ 35us) server_negotiation.cc:407] Connection header received
0809 00:53:05.570986 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:05.570992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:05.571134 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:05.571293 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:05.571999 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.575017 (+ 3018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.579122 (+ 4105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:05.582110 (+ 2988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:05.582405 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:05.583060 (+ 655us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:05.583196 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 00:53:05.583504 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1274,"thread_start_us":875,"threads_started":1}
I20260809 00:53:05.587525 21078 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260809 00:53:05.589154 21121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4293b37228574799b2a8ee7508e605ef. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } } }
I20260809 00:53:05.589633 21121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: This master's current role is: LEARNER
W20260809 00:53:05.879725 21006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:05.880446 21006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:05.880766 21006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:05.896323 21006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:53:05.896642 21006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:05.896823 21006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:05.897001 21006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:53:05.897142 21006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:53:05.915879 21006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:33999,127.9.225.61:37619,127.9.225.60:36845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.60.keytab
--principal=oryx/127.9.225.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.60:36845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.225.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:05.918887 21006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:05.921509 21006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:05.946595 21158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:05.946724 21155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:05.946708 21152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:05.949396 21006 server_base.cc:1034] running on GCE node
I20260809 00:53:05.951396 21006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:05.954610 21006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:05.958169 21006 hybrid_clock.cc:648] HybridClock initialized: now 1786236785958036 us; error 163 us; skew 500 ppm
I20260809 00:53:05.958978 21006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:05 dist-test-slave-j2vl krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236785, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:05.965667 21006 init.cc:377] Logged in from keytab as oryx/127.9.225.60@KRBTEST.COM (short username oryx)
I20260809 00:53:05.968945 21006 webserver.cc:460] Webserver started at http://127.9.225.60:44355/ using document root <none> and password file <none>
I20260809 00:53:05.970604 21006 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:05.970898 21006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:05.971580 21006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:05.977828 21006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data/instance:
uuid: "ce30e401195d44eeb8d6bef722618753"
format_stamp: "Formatted at 2026-08-09 00:53:05 on dist-test-slave-j2vl"
server_key: "87db13be6bc2437c767c739e98720954"
server_key_iv: "f703ae5080ef45f2745033662208c1fa"
server_key_version: "encryptionkey@0"
I20260809 00:53:05.979408 21006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/wal/instance:
uuid: "ce30e401195d44eeb8d6bef722618753"
format_stamp: "Formatted at 2026-08-09 00:53:05 on dist-test-slave-j2vl"
server_key: "87db13be6bc2437c767c739e98720954"
server_key_iv: "f703ae5080ef45f2745033662208c1fa"
server_key_version: "encryptionkey@0"
I20260809 00:53:05.990949 21006 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.009s sys 0.001s
I20260809 00:53:05.998548 21165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:06.001466 21006 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260809 00:53:06.001876 21006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/wal
uuid: "ce30e401195d44eeb8d6bef722618753"
format_stamp: "Formatted at 2026-08-09 00:53:05 on dist-test-slave-j2vl"
server_key: "87db13be6bc2437c767c739e98720954"
server_key_iv: "f703ae5080ef45f2745033662208c1fa"
server_key_version: "encryptionkey@0"
I20260809 00:53:06.002291 21006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:06.025646 21002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } } }
I20260809 00:53:06.026489 21002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:06.029112 21178 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef: attempting to promote NON_VOTER 4e34bc5f7a1944d0b6742a6ad64df4ba to VOTER
I20260809 00:53:06.031261 21178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } }
I20260809 00:53:06.035013 21078 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 LEARNER]: Refusing update from remote peer 4293b37228574799b2a8ee7508e605ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 00:53:06.036123 21176 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 00:53:06.043646 21178 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4e34bc5f7a1944d0b6742a6ad64df4ba (127.9.225.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } }
I20260809 00:53:06.045235 21078 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4e34bc5f7a1944d0b6742a6ad64df4ba (127.9.225.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } }
I20260809 00:53:06.050570 21178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } }
I20260809 00:53:06.051056 21178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:06.051049 21121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } }
I20260809 00:53:06.051591 21121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:53:06.051621 21178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } }
I20260809 00:53:06.051905 21178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:06.059798 21006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:06.068526 21006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:06.069015 21006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:06.106302 21006 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.60:36845
I20260809 00:53:06.106370 21259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.60:36845 every 8 connection(s)
I20260809 00:53:06.109125 21006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
I20260809 00:53:06.114531 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 21006
I20260809 00:53:06.118877 21262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:06.148176 21262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:53:06 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.225.60@KRBTEST.COM
Aug 09 00:53:06 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236785, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.60@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:06.199816 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.119834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.9.225.60:36845)
0809 00:53:06.133136 (+ 13302us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:06.133186 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:06.133385 (+ 199us) server_negotiation.cc:407] Connection header received
0809 00:53:06.143133 (+ 9748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:06.143233 (+ 100us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:06.144672 (+ 1439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:06.145823 (+ 1151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:06.153539 (+ 7716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.158404 (+ 4865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.162516 (+ 4112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.164502 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.165000 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.177519 (+ 12519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:06.177547 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:06.177588 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:06.177717 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:06.187699 (+ 9982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.190241 (+ 2542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.190268 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.190290 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.190743 (+ 453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.191932 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.191946 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.191949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.193181 (+ 1232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:06.194081 (+ 900us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:06.195019 (+ 938us) server_negotiation.cc:299] Negotiation successful
0809 00:53:06.197176 (+ 2157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":12582,"thread_start_us":11972,"threads_started":1}
I20260809 00:53:06.214113 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.154319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:36622)
0809 00:53:06.158090 (+ 3771us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:06.158216 (+ 126us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:06.158832 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:06.159908 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:06.159925 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:06.163721 (+ 3796us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:06.170096 (+ 6375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.170160 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.174680 (+ 4520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.174700 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.179727 (+ 5027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.179777 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.182697 (+ 2920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.182721 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.183298 (+ 577us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.185327 (+ 2029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:06.185414 (+ 87us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:06.193689 (+ 8275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:06.207144 (+ 13455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.207176 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.207193 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.208409 (+ 1216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.209649 (+ 1240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.209653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.209664 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.210149 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.211801 (+ 1652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:06.211810 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:06.212255 (+ 445us) client_negotiation.cc:769] Sending connection context
0809 00:53:06.212616 (+ 361us) client_negotiation.cc:240] Negotiation successful
0809 00:53:06.213172 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"mutex_wait_us":92,"thread_start_us":815,"threads_started":1}
I20260809 00:53:06.215512 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.154603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.9.225.62:33999)
0809 00:53:06.156403 (+ 1800us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:06.156424 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:06.158917 (+ 2493us) server_negotiation.cc:407] Connection header received
0809 00:53:06.159066 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:06.159073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:06.159306 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:06.159620 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:06.170771 (+ 11151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.174272 (+ 3501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.180240 (+ 5968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.182133 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.182471 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.194333 (+ 11862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:06.194345 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:06.194353 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:06.194483 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:06.205064 (+ 10581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.208706 (+ 3642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.208729 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.208749 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.209320 (+ 571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.210401 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.210412 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.210432 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.211562 (+ 1130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:06.211959 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:06.213225 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0809 00:53:06.213891 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1394,"thread_start_us":1110,"threads_started":1}
I20260809 00:53:06.219143 21262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:53:06 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236785, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.60@KRBTEST.COM for oryx/127.9.225.61@KRBTEST.COM
I20260809 00:53:06.258422 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.221667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:37619 (local address 127.0.0.1:54724)
0809 00:53:06.221909 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:06.221954 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:06.222171 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:06.223767 (+ 1596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:06.223774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:06.224713 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:06.225118 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.225133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.227346 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.227352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.229956 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.229999 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.231991 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.232013 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.232738 (+ 725us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.234557 (+ 1819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:06.234638 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:06.242587 (+ 7949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:06.252117 (+ 9530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.252123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.252135 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.253241 (+ 1106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.254201 (+ 960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.254229 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.254249 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.254777 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.256171 (+ 1394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:06.256180 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:06.256600 (+ 420us) client_negotiation.cc:769] Sending connection context
0809 00:53:06.256814 (+ 214us) client_negotiation.cc:240] Negotiation successful
0809 00:53:06.257658 (+ 844us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 00:53:06.258651 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.221548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.9.225.61:37619)
0809 00:53:06.223254 (+ 1706us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:06.223262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:06.223292 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:53:06.223380 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:06.223384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:06.223496 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:06.223715 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:06.225429 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.227035 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.230292 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.231580 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.231909 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.243134 (+ 11225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:06.243167 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:06.243177 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:06.243313 (+ 136us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:06.251754 (+ 8441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.253655 (+ 1901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.253662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.253673 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.254001 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.255031 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.255035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.255037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.255844 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:06.256155 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:06.257094 (+ 939us) server_negotiation.cc:299] Negotiation successful
0809 00:53:06.257541 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1544,"thread_start_us":847,"threads_started":1}
I20260809 00:53:06.261665 21262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:53:06 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236785, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.60@KRBTEST.COM for oryx/127.9.225.60@KRBTEST.COM
I20260809 00:53:06.298537 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.264175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:36845 (local address 127.0.0.1:42052)
0809 00:53:06.264625 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:06.264692 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:06.264963 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:06.265860 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:06.265865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:06.267016 (+ 1151us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:06.267506 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.267520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.270065 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.270078 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.272100 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.272129 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.274343 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.274351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.274807 (+ 456us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.276269 (+ 1462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:06.276320 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:06.281483 (+ 5163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:06.291695 (+ 10212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.291701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.291704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.292913 (+ 1209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.294020 (+ 1107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.294034 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.294036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.294552 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.295914 (+ 1362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:06.295934 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:06.296366 (+ 432us) client_negotiation.cc:769] Sending connection context
0809 00:53:06.297010 (+ 644us) client_negotiation.cc:240] Negotiation successful
0809 00:53:06.297526 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260809 00:53:06.298631 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.264908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.9.225.60:36845)
0809 00:53:06.265134 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:06.265138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:06.265158 (+ 20us) server_negotiation.cc:407] Connection header received
0809 00:53:06.265208 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:06.265212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:06.265282 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:06.265508 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:06.267776 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.269821 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.272376 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.273873 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.274156 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.281927 (+ 7771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:06.281944 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:06.281948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:06.282027 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:06.291266 (+ 9239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.293186 (+ 1920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.293193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.293196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.293780 (+ 584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.294799 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.294816 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.294818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.295473 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:06.295711 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:06.297288 (+ 1577us) server_negotiation.cc:299] Negotiation successful
0809 00:53:06.297779 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 00:53:06.311174 21262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: Bootstrap starting.
I20260809 00:53:06.319561 21262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:06.321748 21262 log.cc:824] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:06.328895 21262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: No bootstrap required, opened a new log
I20260809 00:53:06.348068 21262 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } }
I20260809 00:53:06.349026 21262 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:06.349457 21262 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce30e401195d44eeb8d6bef722618753, State: Initialized, Role: FOLLOWER
I20260809 00:53:06.350889 21262 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } }
I20260809 00:53:06.354096 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } } }
I20260809 00:53:06.355326 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:53:06.356071 21262 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:53:06.371223 21274 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:53:06.372072 21274 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } }
I20260809 00:53:06.379002 20126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce30e401195d44eeb8d6bef722618753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4293b37228574799b2a8ee7508e605ef" is_pre_election: true
I20260809 00:53:06.379940 21078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce30e401195d44eeb8d6bef722618753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" is_pre_election: true
I20260809 00:53:06.378494 21274 leader_election.cc:288] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4293b37228574799b2a8ee7508e605ef (127.9.225.62:33999), 4e34bc5f7a1944d0b6742a6ad64df4ba (127.9.225.61:37619)
I20260809 00:53:06.382351 21167 leader_election.cc:302] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce30e401195d44eeb8d6bef722618753; no voters: 4293b37228574799b2a8ee7508e605ef, 4e34bc5f7a1944d0b6742a6ad64df4ba
I20260809 00:53:06.383581 21274 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 00:53:06.394045 21290 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 00:53:06.394650 21290 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 00:53:06.404265 21006 master_runner.cc:428] Detected that this master ce30e401195d44eeb8d6bef722618753 is joining an existing cluster
I20260809 00:53:06.404690 21006 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.225.60:36845
I20260809 00:53:06.419752 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 20203
Aug 09 00:53:06 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.225.1@KRBTEST.COM
I20260809 00:53:06.490347 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.422327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.9.225.1:46803)
0809 00:53:06.424669 (+ 2342us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:06.424694 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:06.424779 (+ 85us) server_negotiation.cc:407] Connection header received
0809 00:53:06.425023 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:06.425075 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:06.425559 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:06.426587 (+ 1028us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:06.431489 (+ 4902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.436427 (+ 4938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.441578 (+ 5151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.444553 (+ 2975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.446916 (+ 2363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.465800 (+ 18884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:06.465854 (+ 54us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:06.465914 (+ 60us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:06.466046 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:06.477721 (+ 11675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.480087 (+ 2366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.480115 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.480120 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.480657 (+ 537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.481853 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.481867 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.481881 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.483296 (+ 1415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:06.487905 (+ 4609us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:06.488394 (+ 489us) server_negotiation.cc:299] Negotiation successful
0809 00:53:06.489068 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1984,"thread_start_us":1318,"threads_started":1}
I20260809 00:53:06.494349 20411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:06.529727 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.432826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:36638)
0809 00:53:06.450930 (+ 18104us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:06.450985 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:06.451488 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:06.453023 (+ 1535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:06.453033 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:06.455674 (+ 2641us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:06.468170 (+ 12496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.468199 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.472513 (+ 4314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.472532 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.479633 (+ 7101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.479685 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.484260 (+ 4575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.484291 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.484980 (+ 689us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.486792 (+ 1812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:06.486865 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:06.495496 (+ 8631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:06.511010 (+ 15514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.511023 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.511031 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.514192 (+ 3161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.522046 (+ 7854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.522053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.522069 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.522613 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.524367 (+ 1754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:06.524386 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:06.525464 (+ 1078us) client_negotiation.cc:769] Sending connection context
0809 00:53:06.525848 (+ 384us) client_negotiation.cc:240] Negotiation successful
0809 00:53:06.527787 (+ 1939us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17504,"thread_start_us":25066,"threads_started":1}
I20260809 00:53:06.535233 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.432322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.9.225.62:33999)
0809 00:53:06.432771 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:06.432790 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:06.451199 (+ 18409us) server_negotiation.cc:407] Connection header received
0809 00:53:06.451842 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:06.451875 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:06.452263 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:06.452442 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:06.468652 (+ 16210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.470796 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.480138 (+ 9342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.482613 (+ 2475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.483318 (+ 705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.496192 (+ 12874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:06.496228 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:06.496238 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:06.496383 (+ 145us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:06.510605 (+ 14222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.520964 (+ 10359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.521001 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.521007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.521618 (+ 611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.523083 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.523103 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.523117 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.524146 (+ 1029us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:06.527073 (+ 2927us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:06.533077 (+ 6004us) server_negotiation.cc:299] Negotiation successful
0809 00:53:06.533767 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 00:53:06.543186 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.435841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:36845 (local address 127.0.0.1:42066)
0809 00:53:06.458813 (+ 22972us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:06.458875 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:06.459585 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:06.460690 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:06.460698 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:06.467794 (+ 7096us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:06.486613 (+ 18819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.486676 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.490249 (+ 3573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.490257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.495490 (+ 5233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.495523 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.499214 (+ 3691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.499223 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.499634 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.503721 (+ 4087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:06.503767 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:06.510220 (+ 6453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:06.532871 (+ 22651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.532889 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.532893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.534320 (+ 1427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.535446 (+ 1126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.535489 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.535507 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.536252 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.539377 (+ 3125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:06.539383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:06.539919 (+ 536us) client_negotiation.cc:769] Sending connection context
0809 00:53:06.540242 (+ 323us) client_negotiation.cc:240] Negotiation successful
0809 00:53:06.541204 (+ 962us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21636,"thread_start_us":20659,"threads_started":1}
I20260809 00:53:06.546913 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.435257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.9.225.60:36845)
0809 00:53:06.435874 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:06.435894 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:06.459168 (+ 23274us) server_negotiation.cc:407] Connection header received
0809 00:53:06.459952 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:06.459976 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:06.460144 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:06.460341 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:06.487133 (+ 26792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.489715 (+ 2582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.495886 (+ 6171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.498615 (+ 2729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.499001 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.510894 (+ 11893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:06.510904 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:06.510922 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:06.510994 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:06.532535 (+ 21541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.534613 (+ 2078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.534633 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.534637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.535145 (+ 508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.537742 (+ 2597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.537747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.537762 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.539052 (+ 1290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:06.544142 (+ 5090us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:06.544357 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 00:53:06.544833 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":119}
I20260809 00:53:06.551469 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.441956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:37619 (local address 127.0.0.1:54734)
0809 00:53:06.464053 (+ 22097us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:06.464079 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:06.464727 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:06.466458 (+ 1731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:06.466469 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:06.475457 (+ 8988us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:06.489237 (+ 13780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.489275 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.493049 (+ 3774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.493065 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.498556 (+ 5491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.498582 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.508070 (+ 9488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.508082 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.508576 (+ 494us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.518525 (+ 9949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:06.518585 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:06.522727 (+ 4142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:06.535272 (+ 12545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.535285 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.535299 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.536593 (+ 1294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.537674 (+ 1081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.537685 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.537688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.538158 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.544781 (+ 6623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:06.544787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:06.546025 (+ 1238us) client_negotiation.cc:769] Sending connection context
0809 00:53:06.546359 (+ 334us) client_negotiation.cc:240] Negotiation successful
0809 00:53:06.549735 (+ 3376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21613,"mutex_wait_us":31,"thread_start_us":20042,"threads_started":1}
I20260809 00:53:06.548177 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.432524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.9.225.61:37619)
0809 00:53:06.433075 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:06.433081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:06.464447 (+ 31366us) server_negotiation.cc:407] Connection header received
0809 00:53:06.465207 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:06.465228 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:06.465439 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:06.465677 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:06.489744 (+ 24067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.492566 (+ 2822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.500826 (+ 8260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.502547 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.502858 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.523402 (+ 20544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:06.523428 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:06.523437 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:06.523570 (+ 133us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:06.534804 (+ 11234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.536864 (+ 2060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.536883 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.536887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.537324 (+ 437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.538474 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.538482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.538486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.543039 (+ 4553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:06.544882 (+ 1843us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:06.546714 (+ 1832us) server_negotiation.cc:299] Negotiation successful
0809 00:53:06.547298 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295}
W20260809 00:53:06.573449 21185 tablet.cc:2369] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 00:53:06 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236782, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.62@KRBTEST.COM for oryx/127.9.225.60@KRBTEST.COM
I20260809 00:53:06.598528 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.551419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:36845 (local address 127.0.0.1:42080)
0809 00:53:06.554419 (+ 3000us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:06.554467 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:06.554891 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:06.556669 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:06.556696 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:06.558130 (+ 1434us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:06.559369 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.559411 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.562736 (+ 3325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.562765 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.566220 (+ 3455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.566273 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.569210 (+ 2937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.569232 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.569796 (+ 564us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.571675 (+ 1879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:06.571793 (+ 118us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:06.579605 (+ 7812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:06.591636 (+ 12031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.591647 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.591661 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.593229 (+ 1568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.594572 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.594579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.594583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.595078 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.596595 (+ 1517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:06.596601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:06.597021 (+ 420us) client_negotiation.cc:769] Sending connection context
0809 00:53:06.597502 (+ 481us) client_negotiation.cc:240] Negotiation successful
0809 00:53:06.597901 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2735,"thread_start_us":5022,"threads_started":1}
I20260809 00:53:06.600678 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.552724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.9.225.60:36845)
0809 00:53:06.553071 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:06.553081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:06.554625 (+ 1544us) server_negotiation.cc:407] Connection header received
0809 00:53:06.555365 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:06.555382 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:06.555637 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:06.555821 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:06.559816 (+ 3995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.562292 (+ 2476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.566723 (+ 4431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.568810 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.569102 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.580208 (+ 11106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:06.580231 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:06.580237 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:06.580341 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:06.591104 (+ 10763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.593679 (+ 2575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.593691 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.593694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.594234 (+ 540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.595344 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.595353 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.595357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.596328 (+ 971us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:06.596669 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:06.599205 (+ 2536us) server_negotiation.cc:299] Negotiation successful
0809 00:53:06.599746 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260809 00:53:06.605010 20106 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.225.60:36845: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ce30e401195d44eeb8d6bef722618753"
member_type: NON_VOTER
last_known_addr {
host: "127.9.225.60"
port: 36845
}
attrs {
promote: true
}
}
dest_uuid: "4293b37228574799b2a8ee7508e605ef"
cas_config_opid_index: 6
I20260809 00:53:06.606971 20106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true } }
I20260809 00:53:06.613080 21210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:06.614818 21078 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 FOLLOWER]: Refusing update from remote peer 4293b37228574799b2a8ee7508e605ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 00:53:06.617206 21178 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 00:53:06.623521 21210 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 FOLLOWER]: Refusing update from remote peer 4293b37228574799b2a8ee7508e605ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 00:53:06.626817 21178 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 00:53:06.625774 21306 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ce30e401195d44eeb8d6bef722618753 (127.9.225.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true } } }
I20260809 00:53:06.630026 21078 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ce30e401195d44eeb8d6bef722618753 (127.9.225.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true } } }
I20260809 00:53:06.643388 21306 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.225.60:36845
I20260809 00:53:06.644989 21006 master.cc:537] Master@127.9.225.60:36845 shutting down...
I20260809 00:53:06.656689 21311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true } } }
I20260809 00:53:06.658540 21311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:53:06.662281 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true } } }
I20260809 00:53:06.663271 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:06.664544 21178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true } } }
I20260809 00:53:06.665774 21178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:06.675431 21210 raft_consensus.cc:858] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } }
W20260809 00:53:06.675817 21210 raft_consensus.cc:1565] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } }
I20260809 00:53:06.725943 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 20203
I20260809 00:53:06.726797 21210 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4e34bc5f7a1944d0b6742a6ad64df4ba (127.9.225.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } } }
I20260809 00:53:06.844146 21325 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } } }
I20260809 00:53:06.845278 21325 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 00:53:06.845054 21274 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4293b37228574799b2a8ee7508e605ef. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } } }
I20260809 00:53:06.845515 21274 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 00:53:06.846129 21318 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: NON_VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: true } } }
I20260809 00:53:06.847127 21318 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260809 00:53:06.849867 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true } } }
I20260809 00:53:06.850580 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:06.853478 21006 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260809 00:53:06.854251 21006 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260809 00:53:06.854539 21006 tablet_replica.cc:330] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: stopping tablet replica
W20260809 00:53:06.863565 20081 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef -> Peer ce30e401195d44eeb8d6bef722618753 (127.9.225.60:36845): Couldn't send request to peer ce30e401195d44eeb8d6bef722618753. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260809 00:53:06.880118 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:46803
--local_ip_for_outbound_sockets=127.9.225.1
--tserver_master_addrs=127.9.225.62:33999
--webserver_port=36525
--webserver_interface=127.9.225.1
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab
--principal=oryx/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.225.62:33999,127.9.225.61:37619
--tserver_master_addrs=127.9.225.62:33999,127.9.225.61:37619,127.9.225.60:36845 with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:53:06.881426 21006 master.cc:552] Master@127.9.225.60:36845 shutdown complete.
I20260809 00:53:06.881542 21006 master_runner.cc:305] Clearing existing system tablet
I20260809 00:53:06.882169 21006 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:06.882413 21006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:06.888073 21006 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260809 00:53:06.895776 21330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:06.896637 21006 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:06.897199 21006 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.004s
I20260809 00:53:06.897612 21006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/wal
uuid: "ce30e401195d44eeb8d6bef722618753"
format_stamp: "Formatted at 2026-08-09 00:53:05 on dist-test-slave-j2vl"
server_key: "87db13be6bc2437c767c739e98720954"
server_key_iv: "f703ae5080ef45f2745033662208c1fa"
server_key_version: "encryptionkey@0"
I20260809 00:53:06.908670 21006 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:06.927143 21006 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 00:53:06.927735 21006 log.cc:1196] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 00:53:06.929494 21006 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: Deleting consensus metadata
I20260809 00:53:06.930178 21006 master_runner.cc:315] Copying system tablet from 127.9.225.62:33999
I20260809 00:53:06.937674 21006 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: tablet copy: Beginning tablet copy session from remote peer at address 127.9.225.62:33999
I20260809 00:53:06.982707 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.939629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:36648)
0809 00:53:06.943055 (+ 3426us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:06.943175 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:06.943630 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:06.944667 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:06.944699 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:06.946605 (+ 1906us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:06.947851 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.947890 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.951009 (+ 3119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.951021 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.954909 (+ 3888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:06.954958 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.957749 (+ 2791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.957770 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:06.958268 (+ 498us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.959984 (+ 1716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:06.960041 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:06.963162 (+ 3121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:06.974254 (+ 11092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.974272 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.974277 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.975578 (+ 1301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.976645 (+ 1067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:06.976656 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:06.976677 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:06.977109 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:06.979089 (+ 1980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:06.979108 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:06.979760 (+ 652us) client_negotiation.cc:769] Sending connection context
0809 00:53:06.980144 (+ 384us) client_negotiation.cc:240] Negotiation successful
0809 00:53:06.981085 (+ 941us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2976,"thread_start_us":2371,"threads_started":1}
I20260809 00:53:06.984350 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:06.940370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.9.225.62:33999)
0809 00:53:06.941188 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:06.941196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:06.943424 (+ 2228us) server_negotiation.cc:407] Connection header received
0809 00:53:06.944017 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:06.944026 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:06.944218 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:06.944514 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:06.948279 (+ 3765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.950614 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.955365 (+ 4751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:06.957252 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:06.957612 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:06.963916 (+ 6304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:06.963929 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:06.963934 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:06.964085 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:06.973818 (+ 9733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.975907 (+ 2089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.975926 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.975941 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.976368 (+ 427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:06.977384 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:06.977402 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:06.977406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:06.978683 (+ 1277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:06.981641 (+ 2958us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:06.982060 (+ 419us) server_negotiation.cc:299] Negotiation successful
0809 00:53:06.982746 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493}
I20260809 00:53:06.985469 20139 tablet_copy_service.cc:140] P 4293b37228574799b2a8ee7508e605ef: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ce30e401195d44eeb8d6bef722618753 ({username='oryx', principal='oryx/127.9.225.60@KRBTEST.COM'} at 127.0.0.1:36648)
I20260809 00:53:06.985678 20139 tablet_copy_service.cc:161] P 4293b37228574799b2a8ee7508e605ef: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ce30e401195d44eeb8d6bef722618753 at {username='oryx', principal='oryx/127.9.225.60@KRBTEST.COM'} at 127.0.0.1:36648: session id = ce30e401195d44eeb8d6bef722618753-00000000000000000000000000000000
I20260809 00:53:06.993839 20139 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 00:53:06.999043 21006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 00:53:07.013895 21006 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: tablet copy: Starting download of 0 data blocks...
I20260809 00:53:07.014629 21006 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: tablet copy: Starting download of 1 WAL segments...
I20260809 00:53:07.020303 21006 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 00:53:07.033913 20139 tablet_copy_service.cc:342] P 4293b37228574799b2a8ee7508e605ef: Request end of tablet copy session ce30e401195d44eeb8d6bef722618753-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.225.60@KRBTEST.COM'} at 127.0.0.1:36648
I20260809 00:53:07.034482 20139 tablet_copy_service.cc:433] P 4293b37228574799b2a8ee7508e605ef: ending tablet copy session ce30e401195d44eeb8d6bef722618753-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ce30e401195d44eeb8d6bef722618753
I20260809 00:53:07.062186 21006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:07.067615 21341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:07.070712 21344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:07.078449 21351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:07.079892 21006 server_base.cc:1034] running on GCE node
I20260809 00:53:07.080713 21006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:07.082232 21006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:07.083592 21006 hybrid_clock.cc:648] HybridClock initialized: now 1786236787083461 us; error 111 us; skew 500 ppm
I20260809 00:53:07.084158 21006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:07 dist-test-slave-j2vl krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236787, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:07.088966 21006 init.cc:377] Logged in from keytab as oryx/127.9.225.60@KRBTEST.COM (short username oryx)
I20260809 00:53:07.095824 21006 webserver.cc:460] Webserver started at http://127.9.225.60:41295/ using document root <none> and password file <none>
I20260809 00:53:07.096567 21006 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:07.096799 21006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:07.104287 21006 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 00:53:07.108415 21364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:07.109570 21006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 00:53:07.109843 21006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/wal
uuid: "ce30e401195d44eeb8d6bef722618753"
format_stamp: "Formatted at 2026-08-09 00:53:05 on dist-test-slave-j2vl"
server_key: "87db13be6bc2437c767c739e98720954"
server_key_iv: "f703ae5080ef45f2745033662208c1fa"
server_key_version: "encryptionkey@0"
I20260809 00:53:07.110239 21006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:07.134119 21006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:07.143999 21006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:07.200349 21006 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.60:36845
I20260809 00:53:07.200439 21448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.60:36845 every 8 connection(s)
I20260809 00:53:07.202591 21006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
I20260809 00:53:07.212792 21449 sys_catalog.cc:263] Verifying existing consensus state
I20260809 00:53:07.220140 21449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: Bootstrap starting.
I20260809 00:53:07.242116 21178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true } } }
I20260809 00:53:07.242880 21178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:07.272094 21449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 00:53:07.273314 21449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: Bootstrap complete.
I20260809 00:53:07.275404 21449 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true } }
I20260809 00:53:07.275910 21449 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ce30e401195d44eeb8d6bef722618753, State: Initialized, Role: LEARNER
I20260809 00:53:07.276368 21449 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true } }
I20260809 00:53:07.282886 21449 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:53:07.282564 21456 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true } } }
I20260809 00:53:07.284983 21456 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: This master's current role is: LEARNER
I20260809 00:53:07.292037 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:07.242304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:36845 (local address 127.0.0.1:42094)
0809 00:53:07.261739 (+ 19435us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:07.261799 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:07.262156 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:07.263489 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:07.263507 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:07.264565 (+ 1058us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:07.265051 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:07.265080 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:07.269117 (+ 4037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:07.269127 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:07.271496 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:07.271546 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:07.274123 (+ 2577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:07.274132 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:07.274506 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:07.275490 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:07.275539 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:07.277306 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:07.285745 (+ 8439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:07.285759 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:07.285763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:07.286953 (+ 1190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:07.288244 (+ 1291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:07.288251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:07.288266 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:07.288651 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:07.290043 (+ 1392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:07.290061 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:07.290368 (+ 307us) client_negotiation.cc:769] Sending connection context
0809 00:53:07.290682 (+ 314us) client_negotiation.cc:240] Negotiation successful
0809 00:53:07.291270 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19119,"thread_start_us":16307,"threads_started":1}
I20260809 00:53:07.292069 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:07.243374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.9.225.60:36845)
0809 00:53:07.259821 (+ 16447us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:07.259836 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:07.262402 (+ 2566us) server_negotiation.cc:407] Connection header received
0809 00:53:07.262578 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:07.262596 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:07.262787 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:07.263028 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:07.265507 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:07.268595 (+ 3088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:07.271950 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:07.273255 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:07.273608 (+ 353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:07.277776 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:07.277785 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:07.277798 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:07.277867 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:07.285289 (+ 7422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:07.287515 (+ 2226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:07.287521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:07.287524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:07.288011 (+ 487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:07.288872 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:07.288890 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:07.288894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:07.289825 (+ 931us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:07.290058 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:07.290903 (+ 845us) server_negotiation.cc:299] Negotiation successful
0809 00:53:07.291386 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16129,"thread_start_us":15156,"threads_started":1}
I20260809 00:53:07.294697 21405 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260809 00:53:07.296175 21456 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4293b37228574799b2a8ee7508e605ef. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: NON_VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: true } } }
I20260809 00:53:07.296840 21456 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: This master's current role is: LEARNER
I20260809 00:53:07.298449 21307 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef: attempting to promote NON_VOTER ce30e401195d44eeb8d6bef722618753 to VOTER
I20260809 00:53:07.299712 21307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } }
I20260809 00:53:07.305099 21405 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 LEARNER]: Refusing update from remote peer 4293b37228574799b2a8ee7508e605ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 00:53:07.307286 21178 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 00:53:07.313027 21078 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 FOLLOWER]: Refusing update from remote peer 4293b37228574799b2a8ee7508e605ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 00:53:07.315093 21178 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 00:53:07.321084 21306 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ce30e401195d44eeb8d6bef722618753 (127.9.225.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } } }
I20260809 00:53:07.327864 21078 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ce30e401195d44eeb8d6bef722618753 (127.9.225.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } } }
I20260809 00:53:07.345928 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } } }
I20260809 00:53:07.346827 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:07.347669 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } } }
I20260809 00:53:07.348286 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:07.361730 21404 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ce30e401195d44eeb8d6bef722618753 (127.9.225.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } } }
I20260809 00:53:07.371915 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } } }
I20260809 00:53:07.372748 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:07.380858 21456 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } } }
I20260809 00:53:07.381677 21456 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:53:07.390746 21467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4293b37228574799b2a8ee7508e605ef" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } } }
I20260809 00:53:07.391748 21467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:53:07.420881 21471 catalog_manager.cc:1259] Loaded cluster ID: 35e54945d8724482bf33ac4b98a68035
I20260809 00:53:07.421300 21471 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: loading cluster ID for follower catalog manager: success
I20260809 00:53:07.428921 21471 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: acquiring CA information for follower catalog manager: success
I20260809 00:53:07.435216 21471 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 00:53:07.877063 21331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:07.877743 21331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:07.878062 21331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:07.899015 21331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:07.899318 21331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:07.900058 21331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:53:07.923768 21331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab
--principal=oryx/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:46803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=36525
--tserver_master_addrs=127.9.225.62:33999,127.9.225.61:37619,127.9.225.60:36845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:07.926625 21331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:07.929845 21331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:07.965278 21485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:07.970378 21490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:07.970577 21483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:07.971793 21331 server_base.cc:1034] running on GCE node
I20260809 00:53:07.974057 21331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:07.979449 21331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:07.983264 21331 hybrid_clock.cc:648] HybridClock initialized: now 1786236787982859 us; error 152 us; skew 500 ppm
I20260809 00:53:07.984465 21331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:07 dist-test-slave-j2vl krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236787, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:07.993677 21331 init.cc:377] Logged in from keytab as oryx/127.9.225.1@KRBTEST.COM (short username oryx)
I20260809 00:53:07.996915 21331 webserver.cc:460] Webserver started at http://127.9.225.1:36525/ using document root <none> and password file <none>
I20260809 00:53:07.998535 21331 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:07.998970 21331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:08.012960 21331 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.008s sys 0.000s
I20260809 00:53:08.020419 21500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:08.023205 21331 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.003s
I20260809 00:53:08.023649 21331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "14ac5161b2634e11a347324e9b1cb2e5"
format_stamp: "Formatted at 2026-08-09 00:53:03 on dist-test-slave-j2vl"
server_key: "710b28a571fe52fe20c42f6fe25eb740"
server_key_iv: "1ddea19d4880bdac8ac344c72353b10a"
server_key_version: "encryptionkey@0"
I20260809 00:53:08.026710 21331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:08.063705 21331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:08.075083 21331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:08.075826 21331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:08.078382 21331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:08.083200 21331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:08.083659 21331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:08.083987 21331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:08.084277 21331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:08.242266 21331 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:46803
I20260809 00:53:08.242672 21681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:46803 every 8 connection(s)
I20260809 00:53:08.247002 21331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:53:08.250525 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 21331
I20260809 00:53:08.251365 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 20428
Aug 09 00:53:08 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.225.2@KRBTEST.COM
Aug 09 00:53:08 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236787, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.60@KRBTEST.COM
I20260809 00:53:08.372857 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:08.254285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.9.225.2:37159)
0809 00:53:08.267997 (+ 13712us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:08.268009 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:08.268079 (+ 70us) server_negotiation.cc:407] Connection header received
0809 00:53:08.268229 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:08.268244 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:08.268958 (+ 714us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:08.269628 (+ 670us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:08.277013 (+ 7385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.281535 (+ 4522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.286901 (+ 5366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.289482 (+ 2581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.289895 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:08.334863 (+ 44968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:08.334901 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:08.334946 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:08.335060 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:08.347549 (+ 12489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:08.358829 (+ 11280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:08.358850 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:08.358859 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:08.359399 (+ 540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:08.360978 (+ 1579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:08.360986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:08.360989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:08.362198 (+ 1209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:08.363112 (+ 914us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:08.365987 (+ 2875us) server_negotiation.cc:299] Negotiation successful
0809 00:53:08.366502 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13319,"thread_start_us":12797,"threads_started":1}
I20260809 00:53:08.376600 20618 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 09 00:53:08 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236787, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.61@KRBTEST.COM
Aug 09 00:53:08 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236787, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:08.414768 21687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:08.267720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:36845 (local address 127.9.225.1:36245)
0809 00:53:08.287446 (+ 19726us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:08.287542 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:08.303894 (+ 16352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:08.307030 (+ 3136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:08.307061 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:08.309406 (+ 2345us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:08.312879 (+ 3473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:08.312975 (+ 96us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.317978 (+ 5003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.317985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:08.323322 (+ 5337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:08.323363 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.327753 (+ 4390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.327790 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:08.352382 (+ 24592us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:08.354342 (+ 1960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:08.354499 (+ 157us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:08.371717 (+ 17218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:08.383805 (+ 12088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:08.383825 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:08.383874 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:08.385961 (+ 2087us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:08.397701 (+ 11740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:08.397722 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:08.397726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:08.402322 (+ 4596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:08.404354 (+ 2032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:08.404367 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:08.405648 (+ 1281us) client_negotiation.cc:769] Sending connection context
0809 00:53:08.409373 (+ 3725us) client_negotiation.cc:240] Negotiation successful
0809 00:53:08.411131 (+ 1758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18822,"thread_start_us":22516,"threads_started":1}
I20260809 00:53:08.417776 21686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:08.261427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:36245 (local address 127.9.225.60:36845)
0809 00:53:08.286380 (+ 24953us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:08.286399 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:08.289507 (+ 3108us) server_negotiation.cc:407] Connection header received
0809 00:53:08.304811 (+ 15304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:08.304822 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:08.305152 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:08.305326 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:08.313532 (+ 8206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.315993 (+ 2461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.323756 (+ 7763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.325612 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.326049 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:08.372398 (+ 46349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:08.372424 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:08.372436 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:08.372532 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:08.383239 (+ 10707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:08.393293 (+ 10054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:08.393302 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:08.393311 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:08.394015 (+ 704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:08.402745 (+ 8730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:08.402754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:08.402865 (+ 111us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:08.404000 (+ 1135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:08.406689 (+ 2689us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:08.407061 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 00:53:08.407514 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24584,"thread_start_us":22851,"threads_started":1}
I20260809 00:53:08.425647 21682 heartbeater.cc:347] Connected to a master server at 127.9.225.60:36845
I20260809 00:53:08.426539 21682 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:08.429742 21682 heartbeater.cc:511] Master 127.9.225.60:36845 requested a full tablet report, sending...
I20260809 00:53:08.437916 21391 ts_manager.cc:194] Registered new tserver with Master: 14ac5161b2634e11a347324e9b1cb2e5 (127.9.225.1:46803)
I20260809 00:53:08.439332 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:08.305935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.1:44041)
0809 00:53:08.353180 (+ 47245us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:08.353207 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:08.354359 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:08.355827 (+ 1468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:08.355843 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:08.357699 (+ 1856us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:08.371891 (+ 14192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:08.371911 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.382131 (+ 10220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.382142 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:08.389362 (+ 7220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:08.389407 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.392329 (+ 2922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.392339 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:08.392856 (+ 517us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:08.396311 (+ 3455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:08.396377 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:08.413915 (+ 17538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:08.430250 (+ 16335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:08.430267 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:08.430272 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:08.432072 (+ 1800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:08.433819 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:08.433825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:08.433828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:08.434365 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:08.436511 (+ 2146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:08.436531 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:08.436919 (+ 388us) client_negotiation.cc:769] Sending connection context
0809 00:53:08.437246 (+ 327us) client_negotiation.cc:240] Negotiation successful
0809 00:53:08.438025 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27826,"mutex_wait_us":4413,"thread_start_us":28030,"threads_started":1}
I20260809 00:53:08.441992 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:08.306316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:44041 (local address 127.9.225.62:33999)
0809 00:53:08.350434 (+ 44118us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:08.350458 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:08.354649 (+ 4191us) server_negotiation.cc:407] Connection header received
0809 00:53:08.354827 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:08.354833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:08.355220 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:08.355433 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:08.374956 (+ 19523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.379917 (+ 4961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.389895 (+ 9978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.391474 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.391690 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:08.414501 (+ 22811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:08.414526 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:08.414534 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:08.414655 (+ 121us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:08.429874 (+ 15219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:08.432381 (+ 2507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:08.432401 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:08.432407 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:08.432905 (+ 498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:08.434623 (+ 1718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:08.434636 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:08.434643 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:08.436048 (+ 1405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:08.436407 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:08.439998 (+ 3591us) server_negotiation.cc:299] Negotiation successful
0809 00:53:08.441047 (+ 1049us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43787,"thread_start_us":43695,"threads_started":1}
I20260809 00:53:08.443563 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:08.301329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:37619 (local address 127.9.225.1:43223)
0809 00:53:08.330455 (+ 29126us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:08.330513 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:08.331321 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:08.333247 (+ 1926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:08.333278 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:08.354429 (+ 21151us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:08.356536 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:08.356561 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.360826 (+ 4265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.360838 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:08.386382 (+ 25544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:08.386412 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.392822 (+ 6410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.392847 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:08.393562 (+ 715us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:08.395298 (+ 1736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:08.395352 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:08.412306 (+ 16954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:08.430384 (+ 18078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:08.430392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:08.430396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:08.433484 (+ 3088us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:08.435793 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:08.435812 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:08.435817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:08.436430 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:08.438526 (+ 2096us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:08.438532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:08.438818 (+ 286us) client_negotiation.cc:769] Sending connection context
0809 00:53:08.439107 (+ 289us) client_negotiation.cc:240] Negotiation successful
0809 00:53:08.442606 (+ 3499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28658,"mutex_wait_us":9,"thread_start_us":27317,"threads_started":1}
I20260809 00:53:08.443722 21683 heartbeater.cc:347] Connected to a master server at 127.9.225.62:33999
I20260809 00:53:08.444029 21683 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:08.445477 21683 heartbeater.cc:511] Master 127.9.225.62:33999 requested a full tablet report, sending...
I20260809 00:53:08.445765 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:08.300118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:43223 (local address 127.9.225.61:37619)
0809 00:53:08.302005 (+ 1887us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:08.302018 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:08.330822 (+ 28804us) server_negotiation.cc:407] Connection header received
0809 00:53:08.331870 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:08.331881 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:08.332136 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:08.332395 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:08.356949 (+ 24554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.360031 (+ 3082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.389840 (+ 29809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:08.391570 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:08.391867 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:08.414476 (+ 22609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:08.414497 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:08.414503 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:08.414600 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:08.429848 (+ 15248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:08.433806 (+ 3958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:08.433813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:08.433821 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:08.435481 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:08.436765 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:08.436847 (+ 82us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:08.436853 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:08.437964 (+ 1111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:08.444336 (+ 6372us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:08.444582 (+ 246us) server_negotiation.cc:299] Negotiation successful
0809 00:53:08.444921 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":950,"threads_started":1}
I20260809 00:53:08.448849 20106 ts_manager.cc:194] Re-registered known tserver with Master: 14ac5161b2634e11a347324e9b1cb2e5 (127.9.225.1:46803)
I20260809 00:53:08.450599 20106 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:44041
I20260809 00:53:08.452611 21690 heartbeater.cc:347] Connected to a master server at 127.9.225.61:37619
I20260809 00:53:08.452976 21690 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:08.454533 21690 heartbeater.cc:511] Master 127.9.225.61:37619 requested a full tablet report, sending...
I20260809 00:53:08.459489 21061 ts_manager.cc:194] Registered new tserver with Master: 14ac5161b2634e11a347324e9b1cb2e5 (127.9.225.1:46803)
I20260809 00:53:08.582445 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 20428
I20260809 00:53:08.616206 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:37159
--local_ip_for_outbound_sockets=127.9.225.2
--tserver_master_addrs=127.9.225.62:33999
--webserver_port=35055
--webserver_interface=127.9.225.2
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab
--principal=oryx/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.225.62:33999,127.9.225.61:37619
--tserver_master_addrs=127.9.225.62:33999,127.9.225.61:37619,127.9.225.60:36845 with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:53:09.460417 21683 heartbeater.cc:503] Master 127.9.225.62:33999 was elected leader, sending a full tablet report...
W20260809 00:53:09.534441 21698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:09.535135 21698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:09.535503 21698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:09.549896 21698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:09.550151 21698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:09.550379 21698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:53:09.570149 21698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab
--principal=oryx/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:37159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=35055
--tserver_master_addrs=127.9.225.62:33999,127.9.225.61:37619,127.9.225.60:36845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:09.572753 21698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:09.575176 21698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:09.600778 21703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:09.600893 21704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:09.600835 21706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:09.602869 21698 server_base.cc:1034] running on GCE node
I20260809 00:53:09.604055 21698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:09.605969 21698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:09.607719 21698 hybrid_clock.cc:648] HybridClock initialized: now 1786236789607583 us; error 136 us; skew 500 ppm
I20260809 00:53:09.608318 21698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:09 dist-test-slave-j2vl krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236789, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:09.614671 21698 init.cc:377] Logged in from keytab as oryx/127.9.225.2@KRBTEST.COM (short username oryx)
I20260809 00:53:09.617543 21698 webserver.cc:460] Webserver started at http://127.9.225.2:35055/ using document root <none> and password file <none>
I20260809 00:53:09.619112 21698 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:09.619462 21698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:09.631103 21698 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260809 00:53:09.635731 21723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:09.639413 21698 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:53:09.639849 21698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "d4de19738ca547b38df7a46b3eae96a6"
format_stamp: "Formatted at 2026-08-09 00:53:03 on dist-test-slave-j2vl"
server_key: "59334a7203908e313e82b1f6933e19b5"
server_key_iv: "7acc46e7aab26ce47e747dd1ca7b12e1"
server_key_version: "encryptionkey@0"
I20260809 00:53:09.642202 21698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:09.687351 21698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:09.697961 21698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:09.698511 21698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:09.700239 21698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:09.703390 21698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:09.703594 21698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:09.703711 21698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:09.703779 21698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:09.765599 21698 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:37159
I20260809 00:53:09.765795 21894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:37159 every 8 connection(s)
I20260809 00:53:09.768245 21698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:53:09.772951 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 21698
I20260809 00:53:09.773720 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 20635
Aug 09 00:53:09 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236789, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.61@KRBTEST.COM
Aug 09 00:53:09 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236789, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
Aug 09 00:53:09 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.225.3@KRBTEST.COM
Aug 09 00:53:09 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236789, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.60@KRBTEST.COM
I20260809 00:53:09.852793 21901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:09.772748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:33909 (local address 127.9.225.61:37619)
0809 00:53:09.782404 (+ 9656us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:09.782423 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:09.788681 (+ 6258us) server_negotiation.cc:407] Connection header received
0809 00:53:09.797442 (+ 8761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:09.797452 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:09.797654 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:09.797892 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:09.803850 (+ 5958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.807031 (+ 3181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.814708 (+ 7677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.817126 (+ 2418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.817715 (+ 589us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:09.833186 (+ 15471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:09.833204 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:09.833216 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:09.833293 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:09.844145 (+ 10852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:09.846564 (+ 2419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:09.846578 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:09.846582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:09.846895 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:09.848067 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:09.848080 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:09.848083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:09.848643 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:09.848961 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:09.851036 (+ 2075us) server_negotiation.cc:299] Negotiation successful
0809 00:53:09.851546 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9217,"thread_start_us":3365,"threads_started":1}
I20260809 00:53:09.858243 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:09.774372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:37619 (local address 127.9.225.2:33909)
0809 00:53:09.788222 (+ 13850us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:09.788373 (+ 151us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:09.796776 (+ 8403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:09.798566 (+ 1790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:09.798601 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:09.800572 (+ 1971us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:09.803408 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:09.803467 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.807816 (+ 4349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.807829 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:09.814167 (+ 6338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:09.814245 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.818107 (+ 3862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.818133 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:09.819216 (+ 1083us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:09.822766 (+ 3550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:09.822849 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:09.832669 (+ 9820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:09.844535 (+ 11866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:09.844562 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:09.844613 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:09.846165 (+ 1552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:09.847202 (+ 1037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:09.847210 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:09.847218 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:09.847822 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:09.848928 (+ 1106us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:09.848945 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:09.850024 (+ 1079us) client_negotiation.cc:769] Sending connection context
0809 00:53:09.850794 (+ 770us) client_negotiation.cc:240] Negotiation successful
0809 00:53:09.852163 (+ 1369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12970,"thread_start_us":18683,"threads_started":1}
I20260809 00:53:09.861047 21897 heartbeater.cc:347] Connected to a master server at 127.9.225.61:37619
I20260809 00:53:09.861754 21897 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:09.863979 21897 heartbeater.cc:511] Master 127.9.225.61:37619 requested a full tablet report, sending...
I20260809 00:53:09.868279 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:09.776069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.2:45145)
0809 00:53:09.821007 (+ 44938us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:09.821043 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:09.821334 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:09.822735 (+ 1401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:09.822741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:09.824629 (+ 1888us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:09.825838 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:09.825860 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.828778 (+ 2918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.828786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:09.832136 (+ 3350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:09.832161 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.836851 (+ 4690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.836874 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:09.837514 (+ 640us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:09.839045 (+ 1531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:09.839098 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:09.847456 (+ 8358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:09.860537 (+ 13081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:09.860548 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:09.860552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:09.861939 (+ 1387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:09.862683 (+ 744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:09.862688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:09.862694 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:09.863140 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:09.864444 (+ 1304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:09.864454 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:09.864687 (+ 233us) client_negotiation.cc:769] Sending connection context
0809 00:53:09.864916 (+ 229us) client_negotiation.cc:240] Negotiation successful
0809 00:53:09.865588 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44582,"thread_start_us":44124,"threads_started":1}
I20260809 00:53:09.868454 21061 ts_manager.cc:194] Registered new tserver with Master: d4de19738ca547b38df7a46b3eae96a6 (127.9.225.2:37159)
I20260809 00:53:09.869321 21902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:09.774146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:45145 (local address 127.9.225.62:33999)
0809 00:53:09.821891 (+ 47745us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:09.821923 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:09.821980 (+ 57us) server_negotiation.cc:407] Connection header received
0809 00:53:09.822132 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:09.822144 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:09.822352 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:09.822599 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:09.826115 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.827965 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.833188 (+ 5223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.835317 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.835698 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:09.847945 (+ 12247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:09.847957 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:09.847964 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:09.848038 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:09.860165 (+ 12127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:09.862203 (+ 2038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:09.862211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:09.862214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:09.862461 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:09.863384 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:09.863394 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:09.863397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:09.864175 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:09.864366 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:09.866914 (+ 2548us) server_negotiation.cc:299] Negotiation successful
0809 00:53:09.867611 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47355,"thread_start_us":47058,"threads_started":1}
I20260809 00:53:09.871663 21896 heartbeater.cc:347] Connected to a master server at 127.9.225.62:33999
I20260809 00:53:09.872053 21896 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:09.872567 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:09.774760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:38511 (local address 127.9.225.60:36845)
0809 00:53:09.823693 (+ 48933us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:09.823709 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:09.828930 (+ 5221us) server_negotiation.cc:407] Connection header received
0809 00:53:09.829063 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:09.829068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:09.829179 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:09.829324 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:09.833133 (+ 3809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.834573 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.839075 (+ 4502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.840521 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.840835 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:09.855110 (+ 14275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:09.855121 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:09.855126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:09.855188 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:09.863999 (+ 8811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:09.865807 (+ 1808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:09.865814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:09.865823 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:09.866228 (+ 405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:09.867201 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:09.867219 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:09.867234 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:09.868138 (+ 904us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:09.868920 (+ 782us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:09.871440 (+ 2520us) server_negotiation.cc:299] Negotiation successful
0809 00:53:09.871777 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48621,"thread_start_us":49308,"threads_started":1}
I20260809 00:53:09.873677 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:09.776941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:36845 (local address 127.9.225.2:38511)
0809 00:53:09.827683 (+ 50742us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:09.828416 (+ 733us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:09.828747 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:09.829385 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:09.829390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:09.831046 (+ 1656us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:09.832810 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:09.832847 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.834846 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.834858 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:09.838740 (+ 3882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:09.838775 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.840934 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.840942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:09.841689 (+ 747us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:09.843745 (+ 2056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:09.843820 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:09.854657 (+ 10837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:09.864348 (+ 9691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:09.864353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:09.864361 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:09.865588 (+ 1227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:09.866546 (+ 958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:09.866564 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:09.866569 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:09.866955 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:09.868426 (+ 1471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:09.868446 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:09.869549 (+ 1103us) client_negotiation.cc:769] Sending connection context
0809 00:53:09.869798 (+ 249us) client_negotiation.cc:240] Negotiation successful
0809 00:53:09.870525 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50160,"thread_start_us":51201,"threads_started":1}
I20260809 00:53:09.874155 21896 heartbeater.cc:511] Master 127.9.225.62:33999 requested a full tablet report, sending...
I20260809 00:53:09.874253 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:09.776618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.9.225.3:45797)
0809 00:53:09.790035 (+ 13417us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:09.790070 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:09.794654 (+ 4584us) server_negotiation.cc:407] Connection header received
0809 00:53:09.794796 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:09.794805 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:09.795345 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:09.803492 (+ 8147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:09.809033 (+ 5541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.813842 (+ 4809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.822916 (+ 9074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:09.826085 (+ 3169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:09.826481 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:09.850134 (+ 23653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:09.850179 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:09.850220 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:09.850315 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:09.861751 (+ 11436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:09.863549 (+ 1798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:09.863567 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:09.863570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:09.863894 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:09.864790 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:09.864800 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:09.864803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:09.865785 (+ 982us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:09.866385 (+ 600us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:09.872006 (+ 5621us) server_negotiation.cc:299] Negotiation successful
0809 00:53:09.872639 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12849,"thread_start_us":13361,"threads_started":1}
I20260809 00:53:09.875406 21895 heartbeater.cc:347] Connected to a master server at 127.9.225.60:36845
I20260809 00:53:09.875706 21895 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:09.876807 20106 ts_manager.cc:194] Re-registered known tserver with Master: d4de19738ca547b38df7a46b3eae96a6 (127.9.225.2:37159)
I20260809 00:53:09.876971 21895 heartbeater.cc:511] Master 127.9.225.60:36845 requested a full tablet report, sending...
I20260809 00:53:09.878089 20832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:09.879135 20106 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:45145
I20260809 00:53:09.880272 21391 ts_manager.cc:194] Registered new tserver with Master: d4de19738ca547b38df7a46b3eae96a6 (127.9.225.2:37159)
I20260809 00:53:10.015553 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 20635
I20260809 00:53:10.075321 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:45797
--local_ip_for_outbound_sockets=127.9.225.3
--tserver_master_addrs=127.9.225.62:33999
--webserver_port=40767
--webserver_interface=127.9.225.3
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab
--principal=oryx/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.225.62:33999,127.9.225.61:37619
--tserver_master_addrs=127.9.225.62:33999,127.9.225.61:37619,127.9.225.60:36845 with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:10.729094 21915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:10.729732 21915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:10.729944 21915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:10.744702 21915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:10.745007 21915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:10.745374 21915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:53:10.761097 21915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab
--principal=oryx/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:45797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=40767
--tserver_master_addrs=127.9.225.62:33999,127.9.225.61:37619,127.9.225.60:36845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:10.763825 21915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:10.767181 21915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:10.794485 21928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:10.794557 21926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:10.795320 21924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:10.796495 21915 server_base.cc:1034] running on GCE node
I20260809 00:53:10.797622 21915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:10.799405 21915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:10.800909 21915 hybrid_clock.cc:648] HybridClock initialized: now 1786236790800802 us; error 109 us; skew 500 ppm
I20260809 00:53:10.801669 21915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:10 dist-test-slave-j2vl krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236790, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:10.808013 21915 init.cc:377] Logged in from keytab as oryx/127.9.225.3@KRBTEST.COM (short username oryx)
I20260809 00:53:10.811092 21915 webserver.cc:460] Webserver started at http://127.9.225.3:40767/ using document root <none> and password file <none>
I20260809 00:53:10.812852 21915 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:10.813197 21915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:10.824466 21915 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.004s
I20260809 00:53:10.830425 21939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:10.832823 21915 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260809 00:53:10.833178 21915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "0dbc6b27c27b46b094da602fcd3d99c8"
format_stamp: "Formatted at 2026-08-09 00:53:04 on dist-test-slave-j2vl"
server_key: "9ef939bc34bdf600eb11c8e2a25e9213"
server_key_iv: "7e3c256e416208852c15860f5ed5c4c3"
server_key_version: "encryptionkey@0"
I20260809 00:53:10.835247 21915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:10.878896 21915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:10.886250 21896 heartbeater.cc:503] Master 127.9.225.62:33999 was elected leader, sending a full tablet report...
I20260809 00:53:10.888688 21915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:10.889249 21915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:10.891027 21915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:10.893216 21915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:10.893497 21915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:10.893661 21915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:10.893747 21915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:10.956457 21915 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:45797
I20260809 00:53:10.956475 22114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:45797 every 8 connection(s)
I20260809 00:53:10.959831 21915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:53:10.966348 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 21915
Aug 09 00:53:11 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236790, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.61@KRBTEST.COM
Aug 09 00:53:11 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236790, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.60@KRBTEST.COM
Aug 09 00:53:11 dist-test-slave-j2vl krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236790, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:11.085492 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:10.965996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:41025 (local address 127.9.225.60:36845)
0809 00:53:11.013039 (+ 47043us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.013049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.013096 (+ 47us) server_negotiation.cc:407] Connection header received
0809 00:53:11.013227 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.013247 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.013496 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.013793 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:11.020600 (+ 6807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.023061 (+ 2461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.037806 (+ 14745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.039969 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.040231 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.056761 (+ 16530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:11.056777 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:11.056783 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:11.056865 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:11.074251 (+ 17386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.076913 (+ 2662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.076922 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.076927 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.077318 (+ 391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.078990 (+ 1672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.079007 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.079012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.079893 (+ 881us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:11.080229 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.083805 (+ 3576us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.084300 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46794,"thread_start_us":46531,"threads_started":1}
I20260809 00:53:11.085803 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:10.966703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:40165 (local address 127.9.225.61:37619)
0809 00:53:11.004000 (+ 37297us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.004014 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.004055 (+ 41us) server_negotiation.cc:407] Connection header received
0809 00:53:11.006537 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.006567 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.006745 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.006925 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:11.020507 (+ 13582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.023434 (+ 2927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.037180 (+ 13746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.039222 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.039460 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.060707 (+ 21247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:11.060724 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:11.060737 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:11.060838 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:11.074251 (+ 13413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.077491 (+ 3240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.077501 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.077504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.078146 (+ 642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.081802 (+ 3656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.081822 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.081826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.082659 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:11.082948 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.084366 (+ 1418us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.084970 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37054,"thread_start_us":37335,"threads_started":1}
I20260809 00:53:11.089495 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:10.976506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:36845 (local address 127.9.225.3:41025)
0809 00:53:11.002429 (+ 25923us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:11.002482 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:11.005555 (+ 3073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:11.013877 (+ 8322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:11.013900 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:11.015285 (+ 1385us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:11.016931 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.017006 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.023870 (+ 6864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.023880 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.037454 (+ 13574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.037504 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.040990 (+ 3486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.041007 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.041814 (+ 807us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.043913 (+ 2099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:11.044076 (+ 163us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:11.056355 (+ 12279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:11.074765 (+ 18410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:11.074772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:11.074775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:11.076549 (+ 1774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:11.077574 (+ 1025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:11.077587 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:11.077590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:11.078744 (+ 1154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:11.080440 (+ 1696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:11.080465 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:11.081568 (+ 1103us) client_negotiation.cc:769] Sending connection context
0809 00:53:11.082264 (+ 696us) client_negotiation.cc:240] Negotiation successful
0809 00:53:11.085233 (+ 2969us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21194,"mutex_wait_us":2469,"thread_start_us":19043,"threads_started":1}
I20260809 00:53:11.090070 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:10.975819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:37619 (local address 127.9.225.3:40165)
0809 00:53:11.002592 (+ 26773us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:11.002635 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:11.005536 (+ 2901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:11.017162 (+ 11626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:11.017174 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:11.018716 (+ 1542us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:11.020010 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.020058 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.024122 (+ 4064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.024155 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.036880 (+ 12725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.036934 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.040018 (+ 3084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.040045 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.042010 (+ 1965us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.045238 (+ 3228us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:11.045451 (+ 213us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:11.060184 (+ 14733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:11.075900 (+ 15716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:11.075906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:11.075912 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:11.077150 (+ 1238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:11.080916 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:11.080920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:11.080923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:11.081489 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:11.083072 (+ 1583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:11.083077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:11.083491 (+ 414us) client_negotiation.cc:769] Sending connection context
0809 00:53:11.083707 (+ 216us) client_negotiation.cc:240] Negotiation successful
0809 00:53:11.087665 (+ 3958us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21482,"mutex_wait_us":3328,"spinlock_wait_cycles":132992,"thread_start_us":19368,"threads_started":1}
I20260809 00:53:11.092131 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:10.974494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.3:40467)
0809 00:53:11.000727 (+ 26233us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:11.000895 (+ 168us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:11.007240 (+ 6345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:11.008400 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:11.008440 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:11.010727 (+ 2287us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:11.025101 (+ 14374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.025157 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.028814 (+ 3657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.028826 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.038027 (+ 9201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.038075 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.040579 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.040590 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.042708 (+ 2118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.044443 (+ 1735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:11.044513 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:11.056876 (+ 12363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:11.073872 (+ 16996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:11.073891 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:11.073928 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:11.080727 (+ 6799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:11.086153 (+ 5426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:11.086158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:11.086162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:11.086780 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:11.089142 (+ 2362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:11.089150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:11.089592 (+ 442us) client_negotiation.cc:769] Sending connection context
0809 00:53:11.090077 (+ 485us) client_negotiation.cc:240] Negotiation successful
0809 00:53:11.090749 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21358,"mutex_wait_us":1450,"thread_start_us":18347,"threads_started":1}
I20260809 00:53:11.092286 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:10.965807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:40467 (local address 127.9.225.62:33999)
0809 00:53:11.002161 (+ 36354us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.002193 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.002281 (+ 88us) server_negotiation.cc:407] Connection header received
0809 00:53:11.007478 (+ 5197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.007488 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.007614 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.007775 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:11.025527 (+ 17752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.027990 (+ 2463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.038310 (+ 10320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.040055 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.040261 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.057501 (+ 17240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:11.057536 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:11.057543 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:11.057746 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:11.070047 (+ 12301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.083324 (+ 13277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.083344 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.083349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.083783 (+ 434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.087489 (+ 3706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.087501 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.087506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.088719 (+ 1213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:11.089014 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.090371 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.090957 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36077,"thread_start_us":35780,"threads_started":1}
I20260809 00:53:11.095202 22121 heartbeater.cc:347] Connected to a master server at 127.9.225.61:37619
I20260809 00:53:11.095932 22121 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:11.098052 22121 heartbeater.cc:511] Master 127.9.225.61:37619 requested a full tablet report, sending...
I20260809 00:53:11.100030 22117 heartbeater.cc:347] Connected to a master server at 127.9.225.60:36845
I20260809 00:53:11.100433 22117 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:11.101523 22119 heartbeater.cc:347] Connected to a master server at 127.9.225.62:33999
I20260809 00:53:11.101787 22119 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:11.103482 22119 heartbeater.cc:511] Master 127.9.225.62:33999 requested a full tablet report, sending...
I20260809 00:53:11.104147 21061 ts_manager.cc:194] Registered new tserver with Master: 0dbc6b27c27b46b094da602fcd3d99c8 (127.9.225.3:45797)
I20260809 00:53:11.106228 22117 heartbeater.cc:511] Master 127.9.225.60:36845 requested a full tablet report, sending...
I20260809 00:53:11.106494 20106 ts_manager.cc:194] Re-registered known tserver with Master: 0dbc6b27c27b46b094da602fcd3d99c8 (127.9.225.3:45797)
I20260809 00:53:11.109390 20106 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:40467
I20260809 00:53:11.109653 21391 ts_manager.cc:194] Registered new tserver with Master: 0dbc6b27c27b46b094da602fcd3d99c8 (127.9.225.3:45797)
I20260809 00:53:11.113505 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.006764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.9.225.60:36845)
0809 00:53:11.027586 (+ 20822us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.027592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.027659 (+ 67us) server_negotiation.cc:407] Connection header received
0809 00:53:11.027777 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.027785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.028088 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.028242 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:11.049463 (+ 21221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.052010 (+ 2547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.063388 (+ 11378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.065659 (+ 2271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.066041 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.081081 (+ 15040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:11.081094 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:11.081099 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:11.081198 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:11.103629 (+ 22431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.105876 (+ 2247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.105886 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.105889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.106488 (+ 599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.107608 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.107614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.107618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.108466 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:11.111161 (+ 2695us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.111538 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.112041 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20568,"thread_start_us":20202,"threads_started":1}
I20260809 00:53:11.116333 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.006691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.9.225.62:33999)
0809 00:53:11.022222 (+ 15531us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.022242 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.029436 (+ 7194us) server_negotiation.cc:407] Connection header received
0809 00:53:11.029572 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.029699 (+ 127us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.029884 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.030027 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:11.051248 (+ 21221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.053811 (+ 2563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.063425 (+ 9614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.065838 (+ 2413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.066136 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.080183 (+ 14047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:11.080205 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:11.080211 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:11.080330 (+ 119us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:11.100629 (+ 20299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.102910 (+ 2281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.102935 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.102945 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.103776 (+ 831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.111406 (+ 7630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.111425 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.111428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.112343 (+ 915us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:11.113066 (+ 723us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.114326 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.114871 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15270,"thread_start_us":16556,"threads_started":1}
I20260809 00:53:11.123349 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.013242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.9.225.61:37619)
0809 00:53:11.030222 (+ 16980us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.030233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.030256 (+ 23us) server_negotiation.cc:407] Connection header received
0809 00:53:11.030478 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.030489 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.030601 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.030864 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:11.047094 (+ 16230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.053638 (+ 6544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.059578 (+ 5940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.067396 (+ 7818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.067652 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.098223 (+ 30571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:11.098248 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:11.098254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:11.098365 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:11.115295 (+ 16930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.117136 (+ 1841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.117144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.117160 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.117660 (+ 500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.119004 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.119009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.119011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.119760 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:11.120263 (+ 503us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.122001 (+ 1738us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.122416 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16598,"thread_start_us":16906,"threads_started":1}
I20260809 00:53:11.128276 20106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:53:11.150548 20106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:53:11.257200 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.201944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:45797 (local address 127.0.0.1:32950)
0809 00:53:11.218778 (+ 16834us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:11.218829 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:11.219225 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:11.222936 (+ 3711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:11.222944 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:11.222996 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:11.224946 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.224964 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.232786 (+ 7822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.232804 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.248416 (+ 15612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.248438 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.254837 (+ 6399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.254862 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.255386 (+ 524us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.255643 (+ 257us) client_negotiation.cc:769] Sending connection context
0809 00:53:11.255865 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 00:53:11.256105 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16371,"mutex_wait_us":175,"thread_start_us":20508,"threads_started":1}
I20260809 00:53:11.257503 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.202757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.9.225.3:45797)
0809 00:53:11.214126 (+ 11369us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.214149 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.220474 (+ 6325us) server_negotiation.cc:407] Connection header received
0809 00:53:11.220645 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.220659 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.221114 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.221637 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:11.225769 (+ 4132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.230634 (+ 4865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.248660 (+ 18026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.253111 (+ 4451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.253635 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.254491 (+ 856us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.256004 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.256395 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10910,"thread_start_us":10570,"threads_started":1}
I20260809 00:53:11.258836 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.203493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:46803 (local address 127.0.0.1:38550)
0809 00:53:11.219752 (+ 16259us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:11.219859 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:11.220247 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:11.227887 (+ 7640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:11.227893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:11.227980 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:11.229692 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.229713 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.233643 (+ 3930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.233664 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.246060 (+ 12396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.246112 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.255040 (+ 8928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.255059 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.255560 (+ 501us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.256428 (+ 868us) client_negotiation.cc:769] Sending connection context
0809 00:53:11.256577 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 00:53:11.256822 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15451,"mutex_wait_us":767,"thread_start_us":27887,"threads_started":1}
I20260809 00:53:11.259931 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.204067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.9.225.1:46803)
0809 00:53:11.222667 (+ 18600us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.222676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.222736 (+ 60us) server_negotiation.cc:407] Connection header received
0809 00:53:11.225962 (+ 3226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.225985 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.226714 (+ 729us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.227270 (+ 556us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:11.230361 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.233324 (+ 2963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.248943 (+ 15619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.253111 (+ 4168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.255712 (+ 2601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.256263 (+ 551us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.257369 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.259473 (+ 2104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18222,"thread_start_us":21300,"threads_started":1}
I20260809 00:53:11.266055 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.211879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:37159 (local address 127.0.0.1:56886)
0809 00:53:11.232115 (+ 20236us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:11.232145 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:11.232401 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:11.235385 (+ 2984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:11.235412 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:11.235639 (+ 227us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:11.241133 (+ 5494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.241183 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.250703 (+ 9520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.250711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.255492 (+ 4781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.255534 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.263684 (+ 8150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.263707 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.264252 (+ 545us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.264330 (+ 78us) client_negotiation.cc:769] Sending connection context
0809 00:53:11.264502 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 00:53:11.264818 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20035,"mutex_wait_us":156,"thread_start_us":9433,"threads_started":1}
I20260809 00:53:11.267990 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.211498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.9.225.2:37159)
0809 00:53:11.224423 (+ 12925us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.224442 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.232327 (+ 7885us) server_negotiation.cc:407] Connection header received
0809 00:53:11.232776 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.232799 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.233528 (+ 729us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.234151 (+ 623us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:11.246625 (+ 12474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.250252 (+ 3627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.255827 (+ 5575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.261959 (+ 6132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.262330 (+ 371us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.262947 (+ 617us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.266658 (+ 3711us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.267100 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12485,"thread_start_us":12245,"threads_started":1}
I20260809 00:53:11.267119 22018 tablet_service.cc:1467] Processing CreateTablet for tablet bc35f95bc8d14834964497739935da3c (DEFAULT_TABLE table=test-table [id=c5f699a1506b40a08658865e3e4b554e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:11.268803 21582 tablet_service.cc:1467] Processing CreateTablet for tablet bc35f95bc8d14834964497739935da3c (DEFAULT_TABLE table=test-table [id=c5f699a1506b40a08658865e3e4b554e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:11.270504 22018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc35f95bc8d14834964497739935da3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:11.272015 21582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc35f95bc8d14834964497739935da3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:11.277630 21795 tablet_service.cc:1467] Processing CreateTablet for tablet bc35f95bc8d14834964497739935da3c (DEFAULT_TABLE table=test-table [id=c5f699a1506b40a08658865e3e4b554e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:11.281744 21795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc35f95bc8d14834964497739935da3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:11.304435 22184 tablet_bootstrap.cc:492] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5: Bootstrap starting.
I20260809 00:53:11.314198 22185 tablet_bootstrap.cc:492] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8: Bootstrap starting.
I20260809 00:53:11.318025 22184 tablet_bootstrap.cc:654] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:11.322371 22186 tablet_bootstrap.cc:492] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6: Bootstrap starting.
I20260809 00:53:11.324640 22184 log.cc:824] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:11.327955 22185 tablet_bootstrap.cc:654] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:11.332283 22185 log.cc:824] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:11.333956 22186 tablet_bootstrap.cc:654] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:11.334805 22184 tablet_bootstrap.cc:492] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5: No bootstrap required, opened a new log
I20260809 00:53:11.336140 22184 ts_tablet_manager.cc:1397] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.014s
I20260809 00:53:11.339303 22186 log.cc:824] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:11.356491 22186 tablet_bootstrap.cc:492] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6: No bootstrap required, opened a new log
I20260809 00:53:11.357746 22186 ts_tablet_manager.cc:1397] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.011s
I20260809 00:53:11.359733 22185 tablet_bootstrap.cc:492] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8: No bootstrap required, opened a new log
I20260809 00:53:11.360478 22185 ts_tablet_manager.cc:1397] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8: Time spent bootstrapping tablet: real 0.047s user 0.019s sys 0.024s
I20260809 00:53:11.371392 22184 raft_consensus.cc:348] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } } peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } }
I20260809 00:53:11.372855 22184 raft_consensus.cc:374] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:11.373193 22184 raft_consensus.cc:729] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ac5161b2634e11a347324e9b1cb2e5, State: Initialized, Role: FOLLOWER
I20260809 00:53:11.375479 22184 consensus_queue.cc:260] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } } peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } }
I20260809 00:53:11.386077 22184 ts_tablet_manager.cc:1428] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5: Time spent starting tablet: real 0.049s user 0.036s sys 0.010s
I20260809 00:53:11.392105 22186 raft_consensus.cc:348] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } } peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } }
I20260809 00:53:11.393061 22186 raft_consensus.cc:374] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:11.393365 22186 raft_consensus.cc:729] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4de19738ca547b38df7a46b3eae96a6, State: Initialized, Role: FOLLOWER
I20260809 00:53:11.394389 22186 consensus_queue.cc:260] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } } peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } }
I20260809 00:53:11.396386 22185 raft_consensus.cc:348] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } } peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } }
I20260809 00:53:11.397446 22185 raft_consensus.cc:374] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:11.397612 22185 raft_consensus.cc:729] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbc6b27c27b46b094da602fcd3d99c8, State: Initialized, Role: FOLLOWER
I20260809 00:53:11.398592 22185 consensus_queue.cc:260] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } } peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } }
I20260809 00:53:11.400694 22186 ts_tablet_manager.cc:1428] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6: Time spent starting tablet: real 0.043s user 0.024s sys 0.012s
I20260809 00:53:11.408427 22185 ts_tablet_manager.cc:1428] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8: Time spent starting tablet: real 0.048s user 0.018s sys 0.019s
I20260809 00:53:11.409021 22119 heartbeater.cc:503] Master 127.9.225.62:33999 was elected leader, sending a full tablet report...
W20260809 00:53:11.503055 22123 tablet.cc:2369] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:53:11.504421 22194 raft_consensus.cc:482] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:53:11.504886 22194 raft_consensus.cc:504] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } } peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } }
I20260809 00:53:11.506889 22194 leader_election.cc:288] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14ac5161b2634e11a347324e9b1cb2e5 (127.9.225.1:46803), d4de19738ca547b38df7a46b3eae96a6 (127.9.225.2:37159)
I20260809 00:53:11.516556 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.507520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:41045 (local address 127.9.225.1:46803)
0809 00:53:11.507746 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.507753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.507776 (+ 23us) server_negotiation.cc:407] Connection header received
0809 00:53:11.507859 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.507864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.507941 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.508049 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:11.508810 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.510115 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.513598 (+ 3483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.515147 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.515349 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.515490 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.516131 (+ 641us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.516251 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 00:53:11.516805 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.507248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:46803 (local address 127.9.225.3:41045)
0809 00:53:11.507572 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:11.507587 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:11.507657 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:11.508117 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:11.508123 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:11.508203 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:11.508616 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.508633 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.510315 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.510321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.513336 (+ 3015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.513374 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.515489 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.515495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.515885 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.515937 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 00:53:11.516092 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 00:53:11.516318 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":193,"spinlock_wait_cycles":12288}
I20260809 00:53:11.516913 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.507521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:51589 (local address 127.9.225.2:37159)
0809 00:53:11.507746 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.507753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.507776 (+ 23us) server_negotiation.cc:407] Connection header received
0809 00:53:11.507866 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.507870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.507941 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.508049 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:11.508810 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.510217 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.513615 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.515535 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.515652 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.515725 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.516445 (+ 720us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.516627 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 00:53:11.516934 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.507174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:37159 (local address 127.9.225.3:51589)
0809 00:53:11.507442 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:11.507485 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:11.507633 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:11.508117 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:11.508123 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:11.508201 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:11.508643 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.508652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.510382 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.510388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.513352 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:11.513374 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.515778 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.515790 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:11.516246 (+ 456us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.516284 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 00:53:11.516385 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 00:53:11.516563 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":180,"spinlock_wait_cycles":8960}
I20260809 00:53:11.518384 21609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc35f95bc8d14834964497739935da3c" candidate_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ac5161b2634e11a347324e9b1cb2e5" is_pre_election: true
I20260809 00:53:11.518451 21829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc35f95bc8d14834964497739935da3c" candidate_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4de19738ca547b38df7a46b3eae96a6" is_pre_election: true
I20260809 00:53:11.519127 21609 raft_consensus.cc:2454] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbc6b27c27b46b094da602fcd3d99c8 in term 0.
I20260809 00:53:11.519229 21829 raft_consensus.cc:2454] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbc6b27c27b46b094da602fcd3d99c8 in term 0.
I20260809 00:53:11.520264 21943 leader_election.cc:302] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0dbc6b27c27b46b094da602fcd3d99c8, 14ac5161b2634e11a347324e9b1cb2e5, d4de19738ca547b38df7a46b3eae96a6; no voters:
I20260809 00:53:11.520895 22194 raft_consensus.cc:2781] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:53:11.521170 22194 raft_consensus.cc:482] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:53:11.521422 22194 raft_consensus.cc:3037] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:11.524762 22194 raft_consensus.cc:504] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } } peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } }
I20260809 00:53:11.525683 22194 leader_election.cc:288] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [CANDIDATE]: Term 1 election: Requested vote from peers 14ac5161b2634e11a347324e9b1cb2e5 (127.9.225.1:46803), d4de19738ca547b38df7a46b3eae96a6 (127.9.225.2:37159)
I20260809 00:53:11.526386 21609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc35f95bc8d14834964497739935da3c" candidate_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ac5161b2634e11a347324e9b1cb2e5"
I20260809 00:53:11.526494 21829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc35f95bc8d14834964497739935da3c" candidate_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4de19738ca547b38df7a46b3eae96a6"
I20260809 00:53:11.526747 21609 raft_consensus.cc:3037] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:11.526863 21829 raft_consensus.cc:3037] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:11.530609 21609 raft_consensus.cc:2454] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbc6b27c27b46b094da602fcd3d99c8 in term 1.
I20260809 00:53:11.530609 21829 raft_consensus.cc:2454] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbc6b27c27b46b094da602fcd3d99c8 in term 1.
I20260809 00:53:11.531260 21941 leader_election.cc:302] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbc6b27c27b46b094da602fcd3d99c8, d4de19738ca547b38df7a46b3eae96a6; no voters:
I20260809 00:53:11.531695 22194 raft_consensus.cc:2781] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:11.532641 22194 raft_consensus.cc:686] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0dbc6b27c27b46b094da602fcd3d99c8, State: Running, Role: LEADER
I20260809 00:53:11.533571 22194 consensus_queue.cc:237] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } } peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } }
W20260809 00:53:11.534356 21898 tablet.cc:2369] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:53:11.541565 20107 catalog_manager.cc:5495] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dbc6b27c27b46b094da602fcd3d99c8 (127.9.225.3). New cstate: current_term: 1 leader_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } health_report { overall_health: UNKNOWN } } }
I20260809 00:53:11.621640 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.615354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.9.225.3:45797)
0809 00:53:11.615540 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.615545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.615681 (+ 136us) server_negotiation.cc:407] Connection header received
0809 00:53:11.615924 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.615929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.616034 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.616175 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:53:11.616521 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.617211 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.618826 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.619370 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.619486 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.620207 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:53:11.620896 (+ 689us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:53:11.621022 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.621146 (+ 124us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.621228 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 00:53:11.625821 21691 tablet.cc:2369] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:53:11.633215 21609 raft_consensus.cc:1261] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5 [term 1 FOLLOWER]: Refusing update from remote peer 0dbc6b27c27b46b094da602fcd3d99c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:11.633215 21829 raft_consensus.cc:1261] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6 [term 1 FOLLOWER]: Refusing update from remote peer 0dbc6b27c27b46b094da602fcd3d99c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:11.634778 22194 consensus_queue.cc:1035] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:11.635200 22195 consensus_queue.cc:1035] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 00:53:11.670156 22202 mvcc.cc:205] Tried to move back new op lower bound from 7316425898515374080 to 7316425898128351232. Current Snapshot: MvccSnapshot[applied={T|T < 7316425898515374080}]
I20260809 00:53:11.679872 20106 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:11.680279 20106 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:11.689754 20106 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4293b37228574799b2a8ee7508e605ef: Sending DeleteTablet for 3 replicas of tablet bc35f95bc8d14834964497739935da3c
I20260809 00:53:11.691473 22018 tablet_service.cc:1514] Processing DeleteTablet for tablet bc35f95bc8d14834964497739935da3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:11 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:32950
I20260809 00:53:11.691807 21582 tablet_service.cc:1514] Processing DeleteTablet for tablet bc35f95bc8d14834964497739935da3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:11 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:38550
I20260809 00:53:11.691757 21795 tablet_service.cc:1514] Processing DeleteTablet for tablet bc35f95bc8d14834964497739935da3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:11 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:56886
I20260809 00:53:11.707073 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 20056
I20260809 00:53:11.709115 20161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:11.715377 22219 tablet_replica.cc:330] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6: stopping tablet replica
I20260809 00:53:11.716215 22219 raft_consensus.cc:2229] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:11.717068 22219 raft_consensus.cc:2258] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:11.718566 22217 tablet_replica.cc:330] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8: stopping tablet replica
I20260809 00:53:11.719303 22217 raft_consensus.cc:2229] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:53:11.720366 22217 raft_consensus.cc:2258] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:11.721575 22220 tablet_replica.cc:330] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5: stopping tablet replica
I20260809 00:53:11.722295 22220 raft_consensus.cc:2229] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:11.723344 22220 raft_consensus.cc:2258] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:11.726198 22219 ts_tablet_manager.cc:1905] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:11.726203 22217 ts_tablet_manager.cc:1905] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:11.736407 22219 ts_tablet_manager.cc:1918] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:11.736752 22219 log.cc:1196] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/bc35f95bc8d14834964497739935da3c
I20260809 00:53:11.739501 22220 ts_tablet_manager.cc:1905] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:11.744602 22219 ts_tablet_manager.cc:1939] T bc35f95bc8d14834964497739935da3c P d4de19738ca547b38df7a46b3eae96a6: Deleting consensus metadata
I20260809 00:53:11.748451 22217 ts_tablet_manager.cc:1918] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:11.748777 22217 log.cc:1196] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/bc35f95bc8d14834964497739935da3c
I20260809 00:53:11.750481 22220 ts_tablet_manager.cc:1918] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:11.750758 22220 log.cc:1196] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/bc35f95bc8d14834964497739935da3c
I20260809 00:53:11.751726 22220 ts_tablet_manager.cc:1939] T bc35f95bc8d14834964497739935da3c P 14ac5161b2634e11a347324e9b1cb2e5: Deleting consensus metadata
I20260809 00:53:11.757577 22217 ts_tablet_manager.cc:1939] T bc35f95bc8d14834964497739935da3c P 0dbc6b27c27b46b094da602fcd3d99c8: Deleting consensus metadata
I20260809 00:53:11.761858 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.698642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.9.225.61:37619)
0809 00:53:11.719865 (+ 21223us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.719871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.719895 (+ 24us) server_negotiation.cc:407] Connection header received
0809 00:53:11.719969 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.719973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.720043 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.720213 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:11.728854 (+ 8641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.730551 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.733111 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.734186 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.741816 (+ 7630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.749569 (+ 7753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:11.749579 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:11.749585 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:11.749642 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:11.755366 (+ 5724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.756766 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.756771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.756775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.757008 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.757758 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.757764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.757768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.758393 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:11.760611 (+ 2218us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.760814 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.761192 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21067,"thread_start_us":20740,"threads_started":1}
I20260809 00:53:11.784083 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.706244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.9.225.60:36845)
0809 00:53:11.710372 (+ 4128us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.710383 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.759957 (+ 49574us) server_negotiation.cc:407] Connection header received
0809 00:53:11.760154 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.760160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.760289 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.760404 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:11.762901 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.764060 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.766507 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.767469 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.767631 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:11.773968 (+ 6337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:11.773977 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:11.773983 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:11.774042 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:11.779050 (+ 5008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.780305 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.780312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.780331 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.780620 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:11.781254 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:11.781260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:11.781263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:11.781958 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:11.782307 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:11.783125 (+ 818us) server_negotiation.cc:299] Negotiation successful
0809 00:53:11.783547 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3804,"thread_start_us":3514,"threads_started":1}
I20260809 00:53:11.924715 20081 catalog_manager.cc:4841] TS 14ac5161b2634e11a347324e9b1cb2e5 (127.9.225.1:46803): tablet bc35f95bc8d14834964497739935da3c (table test-table [id=c5f699a1506b40a08658865e3e4b554e]) successfully deleted
I20260809 00:53:11.926162 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 20056
I20260809 00:53:11.925861 20079 catalog_manager.cc:4841] TS 0dbc6b27c27b46b094da602fcd3d99c8 (127.9.225.3:45797): tablet bc35f95bc8d14834964497739935da3c (table test-table [id=c5f699a1506b40a08658865e3e4b554e]) successfully deleted
I20260809 00:53:12.010668 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.983735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.9.225.61:37619)
0809 00:53:11.983970 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.983977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.990896 (+ 6919us) server_negotiation.cc:407] Connection header received
0809 00:53:11.991125 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.991131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.991225 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.991334 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:11.994513 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.995669 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.998414 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.999154 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.999320 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:12.003686 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:12.003696 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:12.003703 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:12.003754 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:12.006914 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:12.008094 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:12.008101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:12.008104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:12.008273 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:12.008784 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:12.008789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:12.008793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:12.009266 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:12.009623 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:12.010028 (+ 405us) server_negotiation.cc:299] Negotiation successful
0809 00:53:12.010313 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260809 00:53:12.011111 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:11.977157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.9.225.60:36845)
0809 00:53:11.977451 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:11.977461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:11.990685 (+ 13224us) server_negotiation.cc:407] Connection header received
0809 00:53:11.991014 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:11.991020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:11.991113 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:11.991187 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:11.993992 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.995186 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.998197 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:11.998844 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:11.999009 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:12.002765 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:12.002775 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:12.002780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:12.002833 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:12.006851 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:12.008477 (+ 1626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:12.008483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:12.008488 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:12.008660 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:12.009137 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:12.009143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:12.009146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:12.009583 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:12.009851 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:12.010497 (+ 646us) server_negotiation.cc:299] Negotiation successful
0809 00:53:12.010762 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 00:53:12.669646 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:12.668338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.1:36227)
0809 00:53:12.669160 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:12.669405 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":493,"threads_started":1}
W20260809 00:53:12.670948 21683 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:33999 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
I20260809 00:53:12.672253 22240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:12.669978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.3:48587)
0809 00:53:12.671830 (+ 1852us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:12.672048 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1721,"thread_start_us":1533,"threads_started":1}
W20260809 00:53:12.672835 22119 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:33999 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
I20260809 00:53:12.684729 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:12.683157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.2:46053)
0809 00:53:12.684126 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:12.684443 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":528,"threads_started":1}
W20260809 00:53:12.685396 21896 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:33999 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
I20260809 00:53:13.189556 22242 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4293b37228574799b2a8ee7508e605ef)
I20260809 00:53:13.189833 22242 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } }
I20260809 00:53:13.190726 22243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4293b37228574799b2a8ee7508e605ef)
I20260809 00:53:13.191067 22243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } }
I20260809 00:53:13.190980 22242 leader_election.cc:288] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4293b37228574799b2a8ee7508e605ef (127.9.225.62:33999), 4e34bc5f7a1944d0b6742a6ad64df4ba (127.9.225.61:37619)
I20260809 00:53:13.199632 22243 leader_election.cc:288] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4293b37228574799b2a8ee7508e605ef (127.9.225.62:33999), ce30e401195d44eeb8d6bef722618753 (127.9.225.60:36845)
I20260809 00:53:13.206284 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.193579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:37026)
0809 00:53:13.205686 (+ 12107us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.205928 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11906,"thread_start_us":11649,"threads_started":1}
W20260809 00:53:13.207322 21040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
I20260809 00:53:13.208451 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.191123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:37022)
0809 00:53:13.198991 (+ 7868us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.199303 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7622,"threads_started":1}
W20260809 00:53:13.208745 21366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
I20260809 00:53:13.217236 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.191309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:37619 (local address 127.0.0.1:54770)
0809 00:53:13.206212 (+ 14903us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.206237 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:13.206525 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:13.207197 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:13.207203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:13.207232 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:13.207575 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:13.207588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.209629 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.209638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:13.212743 (+ 3105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:13.212776 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.215827 (+ 3051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.215838 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:13.216158 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.216286 (+ 128us) client_negotiation.cc:769] Sending connection context
0809 00:53:13.216437 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 00:53:13.216668 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14750,"thread_start_us":6837,"threads_started":1}
I20260809 00:53:13.218539 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.191775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.9.225.61:37619)
0809 00:53:13.205183 (+ 13408us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:13.205191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:13.206913 (+ 1722us) server_negotiation.cc:407] Connection header received
0809 00:53:13.206983 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:13.206987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:13.207041 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:13.207172 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:13.207877 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.209398 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.213076 (+ 3678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.215217 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.215448 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.215624 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:13.217615 (+ 1991us) server_negotiation.cc:299] Negotiation successful
0809 00:53:13.217857 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13267,"thread_start_us":13016,"threads_started":1}
I20260809 00:53:13.221549 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.220972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:37048)
0809 00:53:13.221200 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.221311 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 00:53:13.221948 21366 leader_election.cc:334] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4293b37228574799b2a8ee7508e605ef (127.9.225.62:33999): Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
I20260809 00:53:13.223500 21078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce30e401195d44eeb8d6bef722618753" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" is_pre_election: true
I20260809 00:53:13.223839 21078 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce30e401195d44eeb8d6bef722618753 in term 1.
I20260809 00:53:13.224428 21369 leader_election.cc:302] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e34bc5f7a1944d0b6742a6ad64df4ba, ce30e401195d44eeb8d6bef722618753; no voters: 4293b37228574799b2a8ee7508e605ef
I20260809 00:53:13.224793 22242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 00:53:13.224925 22242 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4293b37228574799b2a8ee7508e605ef)
I20260809 00:53:13.225031 22242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 1 FOLLOWER]: Advancing to term 2
I20260809 00:53:13.225939 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.225151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:37064)
0809 00:53:13.225457 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.225671 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 00:53:13.226199 21040 leader_election.cc:334] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4293b37228574799b2a8ee7508e605ef (127.9.225.62:33999): Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
I20260809 00:53:13.228175 22242 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } }
I20260809 00:53:13.228950 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.199981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:36845 (local address 127.0.0.1:42124)
0809 00:53:13.206692 (+ 6711us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.206713 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:13.206838 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:13.221936 (+ 15098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:13.221947 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:13.222002 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:13.222396 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:13.222414 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.224386 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.224391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:13.226526 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:13.226543 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.228009 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.228016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:13.228268 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.228305 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 00:53:13.228423 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 00:53:13.228597 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6597,"thread_start_us":19199,"threads_started":1}
I20260809 00:53:13.229043 22242 leader_election.cc:288] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [CANDIDATE]: Term 2 election: Requested vote from peers 4293b37228574799b2a8ee7508e605ef (127.9.225.62:33999), 4e34bc5f7a1944d0b6742a6ad64df4ba (127.9.225.61:37619)
I20260809 00:53:13.229568 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.207515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.9.225.60:36845)
0809 00:53:13.221309 (+ 13794us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:13.221320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:13.221389 (+ 69us) server_negotiation.cc:407] Connection header received
0809 00:53:13.221506 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:13.221515 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:13.221657 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:13.222585 (+ 928us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:13.222988 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.224230 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.226719 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.227761 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.227942 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.228215 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:13.228451 (+ 236us) server_negotiation.cc:299] Negotiation successful
0809 00:53:13.228605 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13599,"thread_start_us":14622,"threads_started":1}
I20260809 00:53:13.229705 21078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce30e401195d44eeb8d6bef722618753" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba"
I20260809 00:53:13.229964 21078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 1 FOLLOWER]: Advancing to term 2
I20260809 00:53:13.229894 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.229333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:37068)
0809 00:53:13.229595 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.229728 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 00:53:13.230789 21404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ce30e401195d44eeb8d6bef722618753" is_pre_election: true
I20260809 00:53:13.231230 21404 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e34bc5f7a1944d0b6742a6ad64df4ba in current term 2: Already voted for candidate ce30e401195d44eeb8d6bef722618753 in this term.
I20260809 00:53:13.231446 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.230973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:37074)
0809 00:53:13.231191 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.231288 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260809 00:53:13.231659 21366 leader_election.cc:334] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4293b37228574799b2a8ee7508e605ef (127.9.225.62:33999): Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
I20260809 00:53:13.232007 21042 leader_election.cc:302] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e34bc5f7a1944d0b6742a6ad64df4ba; no voters: 4293b37228574799b2a8ee7508e605ef, ce30e401195d44eeb8d6bef722618753
I20260809 00:53:13.233093 21078 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce30e401195d44eeb8d6bef722618753 in term 2.
I20260809 00:53:13.233273 22243 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 00:53:13.233558 21369 leader_election.cc:302] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e34bc5f7a1944d0b6742a6ad64df4ba, ce30e401195d44eeb8d6bef722618753; no voters: 4293b37228574799b2a8ee7508e605ef
I20260809 00:53:13.233915 22242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 00:53:13.234359 22242 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [term 2 LEADER]: Becoming Leader. State: Replica: ce30e401195d44eeb8d6bef722618753, State: Running, Role: LEADER
I20260809 00:53:13.234946 22242 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } }
I20260809 00:53:13.236909 22254 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce30e401195d44eeb8d6bef722618753. Latest consensus state: current_term: 2 leader_uuid: "ce30e401195d44eeb8d6bef722618753" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } } }
I20260809 00:53:13.237290 22254 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:13.238112 22257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:53:13.242563 22257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:53:13.243608 22257 catalog_manager.cc:1259] Loaded cluster ID: 35e54945d8724482bf33ac4b98a68035
I20260809 00:53:13.243717 22257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:53:13.245121 22257 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:53:13.246102 22257 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: Loaded TSK: 0
I20260809 00:53:13.247089 22257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:53:13.284497 21390 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:53:13.292739 21390 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:53:13.297772 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.297300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:37084)
0809 00:53:13.297532 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.297616 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 00:53:13.297999 21078 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [term 2 FOLLOWER]: Refusing update from remote peer ce30e401195d44eeb8d6bef722618753: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260809 00:53:13.299207 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.298747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:37096)
0809 00:53:13.298916 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.299005 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 00:53:13.299029 22254 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260809 00:53:13.299402 21366 consensus_peers.cc:487] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 -> Peer 4293b37228574799b2a8ee7508e605ef (127.9.225.62:33999): Couldn't send request to peer 4293b37228574799b2a8ee7508e605ef. Status: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 00:53:13.302238 22243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce30e401195d44eeb8d6bef722618753. Latest consensus state: current_term: 2 leader_uuid: "ce30e401195d44eeb8d6bef722618753" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } } }
I20260809 00:53:13.302546 22243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:53:13.304741 22242 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ce30e401195d44eeb8d6bef722618753" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } } }
I20260809 00:53:13.305122 22267 mvcc.cc:205] Tried to move back new op lower bound from 7316425905340579840 to 7316425905094438912. Current Snapshot: MvccSnapshot[applied={T|T < 7316425905340579840}]
I20260809 00:53:13.305203 22242 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753 [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:13.305593 22243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ce30e401195d44eeb8d6bef722618753" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4293b37228574799b2a8ee7508e605ef" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 33999 } } peers { permanent_uuid: "4e34bc5f7a1944d0b6742a6ad64df4ba" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 37619 } attrs { promote: false } } peers { permanent_uuid: "ce30e401195d44eeb8d6bef722618753" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 36845 } attrs { promote: false } } }
I20260809 00:53:13.305860 22243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e34bc5f7a1944d0b6742a6ad64df4ba [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:53:13.324600 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.315549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:37159 (local address 127.0.0.1:56898)
0809 00:53:13.316093 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.316116 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:13.316237 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:13.316631 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:13.316639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:13.316700 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:13.316981 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:13.316995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.318695 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.318701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:13.321446 (+ 2745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:13.321476 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.323365 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.323376 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:13.323704 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.323771 (+ 67us) client_negotiation.cc:769] Sending connection context
0809 00:53:13.323861 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 00:53:13.324098 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387}
I20260809 00:53:13.326360 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.315680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.9.225.2:37159)
0809 00:53:13.316236 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:13.316243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:13.316263 (+ 20us) server_negotiation.cc:407] Connection header received
0809 00:53:13.316360 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:13.316365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:13.316455 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:13.317149 (+ 694us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:13.317409 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.318517 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.321764 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.323072 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.325584 (+ 2512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.325761 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:13.325840 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 00:53:13.326034 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":1414,"threads_started":1}
I20260809 00:53:13.327442 21795 tablet_service.cc:1467] Processing CreateTablet for tablet 1275413373dd4f988384009ab9b39c5d (DEFAULT_TABLE table=test-table [id=328a1a909dce49f1866fb1971f3024df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:13.328181 21795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1275413373dd4f988384009ab9b39c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:13.342391 22281 tablet_bootstrap.cc:492] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6: Bootstrap starting.
I20260809 00:53:13.343708 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.316781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:46803 (local address 127.0.0.1:38562)
0809 00:53:13.325123 (+ 8342us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.325152 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:13.325299 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:13.330762 (+ 5463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:13.330771 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:13.330805 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:13.331048 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:13.331059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.334830 (+ 3771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.334837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:13.337500 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:13.337542 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.342593 (+ 5051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.342600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:13.342900 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.342944 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 00:53:13.343026 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 00:53:13.343197 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8173,"thread_start_us":12849,"threads_started":1}
I20260809 00:53:13.344563 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.316163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:45797 (local address 127.0.0.1:32966)
0809 00:53:13.316671 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.316694 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:13.316830 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:13.334847 (+ 18017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:13.334853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:13.334893 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:13.335259 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:13.335273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.337947 (+ 2674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.337960 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:13.340984 (+ 3024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:13.341016 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.343334 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.343343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:13.343767 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.343829 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 00:53:13.343946 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 00:53:13.344244 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344}
I20260809 00:53:13.345053 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.317216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.9.225.1:46803)
0809 00:53:13.330306 (+ 13090us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:13.330315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:13.330344 (+ 29us) server_negotiation.cc:407] Connection header received
0809 00:53:13.330435 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:13.330441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:13.330544 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:13.331200 (+ 656us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:13.331383 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.334589 (+ 3206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.337998 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.342320 (+ 4322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.344322 (+ 2002us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.344471 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:13.344542 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 00:53:13.344725 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12911,"thread_start_us":12790,"threads_started":1}
I20260809 00:53:13.345506 22281 tablet_bootstrap.cc:654] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:13.346024 21582 tablet_service.cc:1467] Processing CreateTablet for tablet 1275413373dd4f988384009ab9b39c5d (DEFAULT_TABLE table=test-table [id=328a1a909dce49f1866fb1971f3024df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:13.346519 21582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1275413373dd4f988384009ab9b39c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:13.346560 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.316525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.9.225.3:45797)
0809 00:53:13.334442 (+ 17917us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:13.334449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:13.334475 (+ 26us) server_negotiation.cc:407] Connection header received
0809 00:53:13.334548 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:13.334554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:13.334634 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:13.335523 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:13.335826 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.337620 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.341367 (+ 3747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.342861 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.345057 (+ 2196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.345223 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:13.345292 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 00:53:13.345530 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17617,"thread_start_us":17730,"threads_started":1}
I20260809 00:53:13.347913 22018 tablet_service.cc:1467] Processing CreateTablet for tablet 1275413373dd4f988384009ab9b39c5d (DEFAULT_TABLE table=test-table [id=328a1a909dce49f1866fb1971f3024df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:13.348423 22018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1275413373dd4f988384009ab9b39c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:13.351553 22281 tablet_bootstrap.cc:492] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6: No bootstrap required, opened a new log
I20260809 00:53:13.351830 22281 ts_tablet_manager.cc:1397] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260809 00:53:13.353163 22281 raft_consensus.cc:348] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } } peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } }
I20260809 00:53:13.353760 22281 raft_consensus.cc:374] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:13.353931 22281 raft_consensus.cc:729] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4de19738ca547b38df7a46b3eae96a6, State: Initialized, Role: FOLLOWER
I20260809 00:53:13.354264 22281 consensus_queue.cc:260] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } } peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } }
I20260809 00:53:13.355315 22281 ts_tablet_manager.cc:1428] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 00:53:13.360452 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.355649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.2:57935)
0809 00:53:13.359758 (+ 4109us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.360011 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3861,"thread_start_us":3598,"threads_started":1}
I20260809 00:53:13.362668 22287 tablet_bootstrap.cc:492] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5: Bootstrap starting.
I20260809 00:53:13.365728 21895 heartbeater.cc:503] Master 127.9.225.60:36845 was elected leader, sending a full tablet report...
I20260809 00:53:13.367863 22287 tablet_bootstrap.cc:654] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:13.370040 22288 tablet_bootstrap.cc:492] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8: Bootstrap starting.
I20260809 00:53:13.372586 22288 tablet_bootstrap.cc:654] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:13.372948 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.372060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.2:53505)
0809 00:53:13.372623 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.372739 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260809 00:53:13.374200 22287 tablet_bootstrap.cc:492] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5: No bootstrap required, opened a new log
I20260809 00:53:13.374440 22287 ts_tablet_manager.cc:1397] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260809 00:53:13.375444 22288 tablet_bootstrap.cc:492] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8: No bootstrap required, opened a new log
I20260809 00:53:13.375639 22288 ts_tablet_manager.cc:1397] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 00:53:13.375497 22287 raft_consensus.cc:348] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } } peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } }
I20260809 00:53:13.375795 22287 raft_consensus.cc:374] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:13.375921 22287 raft_consensus.cc:729] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ac5161b2634e11a347324e9b1cb2e5, State: Initialized, Role: FOLLOWER
I20260809 00:53:13.376202 22287 consensus_queue.cc:260] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } } peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } }
I20260809 00:53:13.376550 22288 raft_consensus.cc:348] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } } peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } }
I20260809 00:53:13.376850 22288 raft_consensus.cc:374] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:13.376955 22288 raft_consensus.cc:729] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dbc6b27c27b46b094da602fcd3d99c8, State: Initialized, Role: FOLLOWER
I20260809 00:53:13.377180 22287 ts_tablet_manager.cc:1428] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 00:53:13.377254 22288 consensus_queue.cc:260] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } } peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } }
I20260809 00:53:13.381702 21682 heartbeater.cc:503] Master 127.9.225.60:36845 was elected leader, sending a full tablet report...
I20260809 00:53:13.382164 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.378460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.1:37029)
0809 00:53:13.381623 (+ 3163us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.381826 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2983,"thread_start_us":3168,"threads_started":1}
I20260809 00:53:13.385004 22288 ts_tablet_manager.cc:1428] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260809 00:53:13.389211 22297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.384185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.1:53685)
0809 00:53:13.385943 (+ 1758us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.386429 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1599,"threads_started":1}
I20260809 00:53:13.389977 22117 heartbeater.cc:503] Master 127.9.225.60:36845 was elected leader, sending a full tablet report...
I20260809 00:53:13.394634 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.378911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.3:33541)
0809 00:53:13.393988 (+ 15077us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.394280 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14911,"thread_start_us":14731,"threads_started":1}
I20260809 00:53:13.396764 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.396145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.3:45013)
0809 00:53:13.396444 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.396572 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260809 00:53:13.465786 22292 raft_consensus.cc:482] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:53:13.466080 22292 raft_consensus.cc:504] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } } peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } }
I20260809 00:53:13.467185 22292 leader_election.cc:288] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4de19738ca547b38df7a46b3eae96a6 (127.9.225.2:37159), 14ac5161b2634e11a347324e9b1cb2e5 (127.9.225.1:46803)
I20260809 00:53:13.468036 21829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1275413373dd4f988384009ab9b39c5d" candidate_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4de19738ca547b38df7a46b3eae96a6" is_pre_election: true
I20260809 00:53:13.468007 21609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1275413373dd4f988384009ab9b39c5d" candidate_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ac5161b2634e11a347324e9b1cb2e5" is_pre_election: true
I20260809 00:53:13.468331 21829 raft_consensus.cc:2454] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbc6b27c27b46b094da602fcd3d99c8 in term 0.
I20260809 00:53:13.468331 21609 raft_consensus.cc:2454] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dbc6b27c27b46b094da602fcd3d99c8 in term 0.
I20260809 00:53:13.468952 21941 leader_election.cc:302] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dbc6b27c27b46b094da602fcd3d99c8, d4de19738ca547b38df7a46b3eae96a6; no voters:
I20260809 00:53:13.469368 22292 raft_consensus.cc:2781] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:53:13.469553 22292 raft_consensus.cc:482] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:53:13.469663 22292 raft_consensus.cc:3037] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:13.472321 22292 raft_consensus.cc:504] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } } peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } }
I20260809 00:53:13.473145 22292 leader_election.cc:288] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [CANDIDATE]: Term 1 election: Requested vote from peers d4de19738ca547b38df7a46b3eae96a6 (127.9.225.2:37159), 14ac5161b2634e11a347324e9b1cb2e5 (127.9.225.1:46803)
I20260809 00:53:13.473918 21829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1275413373dd4f988384009ab9b39c5d" candidate_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4de19738ca547b38df7a46b3eae96a6"
I20260809 00:53:13.473974 21609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1275413373dd4f988384009ab9b39c5d" candidate_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14ac5161b2634e11a347324e9b1cb2e5"
I20260809 00:53:13.474181 21829 raft_consensus.cc:3037] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:13.474308 21609 raft_consensus.cc:3037] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:13.476665 21609 raft_consensus.cc:2454] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbc6b27c27b46b094da602fcd3d99c8 in term 1.
I20260809 00:53:13.476661 21829 raft_consensus.cc:2454] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dbc6b27c27b46b094da602fcd3d99c8 in term 1.
I20260809 00:53:13.477398 21943 leader_election.cc:302] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0dbc6b27c27b46b094da602fcd3d99c8, 14ac5161b2634e11a347324e9b1cb2e5, d4de19738ca547b38df7a46b3eae96a6; no voters:
I20260809 00:53:13.477958 22292 raft_consensus.cc:2781] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:13.478767 22292 raft_consensus.cc:686] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0dbc6b27c27b46b094da602fcd3d99c8, State: Running, Role: LEADER
I20260809 00:53:13.479444 22292 consensus_queue.cc:237] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } } peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } }
I20260809 00:53:13.480356 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.479782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.3:35191)
0809 00:53:13.479970 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.480133 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 00:53:13.483613 21390 catalog_manager.cc:5495] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dbc6b27c27b46b094da602fcd3d99c8 (127.9.225.3). New cstate: current_term: 1 leader_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dbc6b27c27b46b094da602fcd3d99c8" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 45797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 } health_report { overall_health: UNKNOWN } } }
I20260809 00:53:13.530445 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.525164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.9.225.3:45797)
0809 00:53:13.525489 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:13.525494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:13.525513 (+ 19us) server_negotiation.cc:407] Connection header received
0809 00:53:13.525603 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:13.525607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:13.525682 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:13.525764 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:53:13.526126 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.526985 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.528505 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.529005 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.529111 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.529592 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:53:13.529719 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:53:13.529832 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:13.530005 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 00:53:13.530083 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 00:53:13.533982 21609 raft_consensus.cc:1261] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5 [term 1 FOLLOWER]: Refusing update from remote peer 0dbc6b27c27b46b094da602fcd3d99c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:13.534057 21829 raft_consensus.cc:1261] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6 [term 1 FOLLOWER]: Refusing update from remote peer 0dbc6b27c27b46b094da602fcd3d99c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:13.534778 22299 consensus_queue.cc:1035] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14ac5161b2634e11a347324e9b1cb2e5" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:13.535159 22292 consensus_queue.cc:1035] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4de19738ca547b38df7a46b3eae96a6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:13.538004 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.537176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.1:47023)
0809 00:53:13.537496 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.537598 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260809 00:53:13.538506 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.537775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.2:33099)
0809 00:53:13.538075 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.538224 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260809 00:53:13.550920 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.550150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.1:54359)
0809 00:53:13.550450 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.550615 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260809 00:53:13.552340 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.551663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.2:60009)
0809 00:53:13.551977 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.552099 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260809 00:53:13.561025 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.558822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.3:51455)
0809 00:53:13.560305 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.560422 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1362}
I20260809 00:53:13.562937 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.561710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.9.225.3:60307)
0809 00:53:13.562585 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.562694 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":756,"threads_started":1}
I20260809 00:53:13.566233 21390 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:13.566479 21390 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:13.573941 21390 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce30e401195d44eeb8d6bef722618753: Sending DeleteTablet for 3 replicas of tablet 1275413373dd4f988384009ab9b39c5d
I20260809 00:53:13.575685 22018 tablet_service.cc:1514] Processing DeleteTablet for tablet 1275413373dd4f988384009ab9b39c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:13 UTC) from {username='oryx', principal='oryx/127.9.225.60@KRBTEST.COM'} at 127.0.0.1:32966
I20260809 00:53:13.575693 21582 tablet_service.cc:1514] Processing DeleteTablet for tablet 1275413373dd4f988384009ab9b39c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:13 UTC) from {username='oryx', principal='oryx/127.9.225.60@KRBTEST.COM'} at 127.0.0.1:38562
I20260809 00:53:13.576292 21795 tablet_service.cc:1514] Processing DeleteTablet for tablet 1275413373dd4f988384009ab9b39c5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:13 UTC) from {username='oryx', principal='oryx/127.9.225.60@KRBTEST.COM'} at 127.0.0.1:56898
I20260809 00:53:13.576579 22320 tablet_replica.cc:330] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5: stopping tablet replica
I20260809 00:53:13.576843 22320 raft_consensus.cc:2229] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:13.577236 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 21331
I20260809 00:53:13.577694 22321 tablet_replica.cc:330] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6: stopping tablet replica
I20260809 00:53:13.578227 22318 tablet_replica.cc:330] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8: stopping tablet replica
I20260809 00:53:13.578536 22318 raft_consensus.cc:2229] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:53:13.579124 22318 raft_consensus.cc:2258] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:13.579599 22320 raft_consensus.cc:2258] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:13.580579 22318 ts_tablet_manager.cc:1905] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:13.582288 22320 ts_tablet_manager.cc:1905] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:13.584504 22321 raft_consensus.cc:2229] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:13.584898 22321 raft_consensus.cc:2258] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:13.586884 22321 ts_tablet_manager.cc:1905] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:13.587932 22318 ts_tablet_manager.cc:1918] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:13.588099 22318 log.cc:1196] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/1275413373dd4f988384009ab9b39c5d
I20260809 00:53:13.588600 22318 ts_tablet_manager.cc:1939] T 1275413373dd4f988384009ab9b39c5d P 0dbc6b27c27b46b094da602fcd3d99c8: Deleting consensus metadata
I20260809 00:53:13.589641 22320 ts_tablet_manager.cc:1918] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:13.589800 22320 log.cc:1196] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/1275413373dd4f988384009ab9b39c5d
I20260809 00:53:13.590122 21365 catalog_manager.cc:4841] TS 0dbc6b27c27b46b094da602fcd3d99c8 (127.9.225.3:45797): tablet 1275413373dd4f988384009ab9b39c5d (table test-table [id=328a1a909dce49f1866fb1971f3024df]) successfully deleted
I20260809 00:53:13.590363 22320 ts_tablet_manager.cc:1939] T 1275413373dd4f988384009ab9b39c5d P 14ac5161b2634e11a347324e9b1cb2e5: Deleting consensus metadata
I20260809 00:53:13.591493 21368 catalog_manager.cc:4841] TS 14ac5161b2634e11a347324e9b1cb2e5 (127.9.225.1:46803): tablet 1275413373dd4f988384009ab9b39c5d (table test-table [id=328a1a909dce49f1866fb1971f3024df]) successfully deleted
I20260809 00:53:13.595410 22321 ts_tablet_manager.cc:1918] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:13.595602 22321 log.cc:1196] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/1275413373dd4f988384009ab9b39c5d
I20260809 00:53:13.596052 22321 ts_tablet_manager.cc:1939] T 1275413373dd4f988384009ab9b39c5d P d4de19738ca547b38df7a46b3eae96a6: Deleting consensus metadata
I20260809 00:53:13.597214 21366 catalog_manager.cc:4841] TS d4de19738ca547b38df7a46b3eae96a6 (127.9.225.2:37159): tablet 1275413373dd4f988384009ab9b39c5d (table test-table [id=328a1a909dce49f1866fb1971f3024df]) successfully deleted
I20260809 00:53:13.612862 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.578771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.9.225.1:46803)
0809 00:53:13.578997 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:13.579007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:13.592014 (+ 13007us) server_negotiation.cc:407] Connection header received
0809 00:53:13.592137 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:13.592141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:13.592247 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:13.592457 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:13.594451 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.595193 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.596765 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.597440 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.597585 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.603976 (+ 6391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:13.604008 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:13.604048 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:13.604116 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:13.608295 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:13.609845 (+ 1550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:13.609859 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:13.609863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:13.610089 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:13.610612 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:13.610617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:13.610620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:13.611458 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:13.611922 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:13.612299 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 00:53:13.612543 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 00:53:13.613677 21675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:13.728972 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 21331
I20260809 00:53:13.761463 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 21698
I20260809 00:53:13.783097 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.762448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.9.225.2:37159)
0809 00:53:13.763000 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:13.763005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:13.763021 (+ 16us) server_negotiation.cc:407] Connection header received
0809 00:53:13.763075 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:13.763079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:13.763149 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:13.763265 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:13.765166 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.766162 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.767571 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.768122 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.768291 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.771689 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:13.771716 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:13.771744 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:13.771793 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:13.777347 (+ 5554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:13.778682 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:13.778696 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:13.778702 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:13.778920 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:13.779435 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:13.779442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:13.779447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:13.780122 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:13.780567 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:13.782234 (+ 1667us) server_negotiation.cc:299] Negotiation successful
0809 00:53:13.782555 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415}
I20260809 00:53:13.784605 21890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:13.866151 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.864898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:37112)
0809 00:53:13.865805 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.865922 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":660,"threads_started":1}
I20260809 00:53:13.867779 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.867364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:33999 (local address 127.0.0.1:37118)
0809 00:53:13.867530 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:13.867632 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 00:53:13.901263 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 21698
I20260809 00:53:13.929800 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 21915
I20260809 00:53:13.945583 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:13.930758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.9.225.3:45797)
0809 00:53:13.931029 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:13.931036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:13.931151 (+ 115us) server_negotiation.cc:407] Connection header received
0809 00:53:13.931324 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:13.931328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:13.931403 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:13.931497 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:13.932933 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.933942 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.935266 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:13.935680 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:13.935788 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:13.938710 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:13.938729 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:13.938748 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:13.938821 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:13.942106 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:13.943132 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:13.943143 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:13.943145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:13.943315 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:13.943740 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:13.943744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:13.943746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:13.944270 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:13.944517 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:13.944902 (+ 385us) server_negotiation.cc:299] Negotiation successful
0809 00:53:13.945101 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 00:53:13.946324 22108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:14.076061 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 21915
I20260809 00:53:14.114673 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 20847
I20260809 00:53:14.116020 21110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:14.251408 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 20847
I20260809 00:53:14.288954 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 21006
I20260809 00:53:14.290163 21440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:14.428635 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 21006
2026-08-09T00:53:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (14257 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:53:14 dist-test-slave-j2vl krb5kdc[22341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:53:14 dist-test-slave-j2vl krb5kdc[22341](info): set up 2 sockets
Aug 09 00:53:14 dist-test-slave-j2vl krb5kdc[22341](info): commencing operation
krb5kdc: starting...
W20260809 00:53:15.864341 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.370s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:53:16 dist-test-slave-j2vl krb5kdc[22341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:53:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.62.keytab.
Entry for principal oryx/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.62.keytab.
I20260809 00:53:16.177801 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:41961
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.62.keytab
--principal=oryx/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:41961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:16.559764 22368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:16.560374 22368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:16.560534 22368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:16.568293 22368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:53:16.568447 22368 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 00:53:16.568521 22368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:16.568620 22368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:16.568720 22368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:53:16.568847 22368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:53:16.579629 22368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:41961
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.62.keytab
--principal=oryx/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:41961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:16.581290 22368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:16.582823 22368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:16.596616 22376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:16.596722 22380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:16.596640 22375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:16.597728 22368 server_base.cc:1034] running on GCE node
I20260809 00:53:16.598744 22368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:16.600472 22368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:16.601753 22368 hybrid_clock.cc:648] HybridClock initialized: now 1786236796601680 us; error 52 us; skew 500 ppm
I20260809 00:53:16.602166 22368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:16 dist-test-slave-j2vl krb5kdc[22341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236796, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:16.605993 22368 init.cc:377] Logged in from keytab as oryx/127.9.225.62@KRBTEST.COM (short username oryx)
I20260809 00:53:16.607903 22368 webserver.cc:460] Webserver started at http://127.9.225.62:37121/ using document root <none> and password file <none>
I20260809 00:53:16.608794 22368 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:16.608927 22368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:16.609483 22368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:16.613152 22368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "1cce8029d33641c296e7c92ae5be7448"
format_stamp: "Formatted at 2026-08-09 00:53:16 on dist-test-slave-j2vl"
server_key: "0c688c41e39f43e60f9924269848c092"
server_key_iv: "1a4738bbe785af57dcf02bd231813548"
server_key_version: "encryptionkey@0"
I20260809 00:53:16.614255 22368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "1cce8029d33641c296e7c92ae5be7448"
format_stamp: "Formatted at 2026-08-09 00:53:16 on dist-test-slave-j2vl"
server_key: "0c688c41e39f43e60f9924269848c092"
server_key_iv: "1a4738bbe785af57dcf02bd231813548"
server_key_version: "encryptionkey@0"
I20260809 00:53:16.620625 22368 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 00:53:16.625051 22395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:16.626833 22368 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:53:16.627102 22368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "1cce8029d33641c296e7c92ae5be7448"
format_stamp: "Formatted at 2026-08-09 00:53:16 on dist-test-slave-j2vl"
server_key: "0c688c41e39f43e60f9924269848c092"
server_key_iv: "1a4738bbe785af57dcf02bd231813548"
server_key_version: "encryptionkey@0"
I20260809 00:53:16.627341 22368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:16.648767 22368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:16.654129 22368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:16.654479 22368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:16.681071 22368 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:41961
I20260809 00:53:16.681102 22486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:41961 every 8 connection(s)
I20260809 00:53:16.682809 22368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:53:16.687345 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:16.690863 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 22368
I20260809 00:53:16.691133 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:53:16.692028 10116 external_mini_cluster.cc:1428] Setting key 2642a66bc9b569cc25b30e0cb262eab8
I20260809 00:53:16.703519 22487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448: Bootstrap starting.
I20260809 00:53:16.708469 22487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:16.710062 22487 log.cc:824] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448: Log is configured to *not* fsync() on all Append() calls
Aug 09 00:53:16 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:16.713640 22487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448: No bootstrap required, opened a new log
I20260809 00:53:16.720544 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:16.694662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.9.225.62:41961)
0809 00:53:16.695798 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:16.695838 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:16.695886 (+ 48us) server_negotiation.cc:407] Connection header received
0809 00:53:16.698316 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:16.698367 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:16.699022 (+ 655us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:16.699471 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:16.701631 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:16.703090 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:16.705076 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:16.706018 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:16.706206 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:16.711579 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:16.711590 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:16.711610 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:16.711661 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:16.715185 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:16.716099 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:16.716108 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:16.716120 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:16.716335 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:16.716832 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:16.716836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:16.716839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:16.717633 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:16.718028 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:16.718692 (+ 664us) server_negotiation.cc:299] Negotiation successful
0809 00:53:16.719256 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":638,"threads_started":1}
I20260809 00:53:16.723205 22487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cce8029d33641c296e7c92ae5be7448" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41961 } }
I20260809 00:53:16.723794 22487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:16.723928 22487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cce8029d33641c296e7c92ae5be7448, State: Initialized, Role: FOLLOWER
I20260809 00:53:16.724710 22487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cce8029d33641c296e7c92ae5be7448" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41961 } }
I20260809 00:53:16.725052 22487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:53:16.725210 22487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:53:16.725430 22487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:16.728227 22487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cce8029d33641c296e7c92ae5be7448" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41961 } }
I20260809 00:53:16.728957 22487 leader_election.cc:302] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cce8029d33641c296e7c92ae5be7448; no voters:
I20260809 00:53:16.729889 22487 leader_election.cc:288] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:53:16.730052 22501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:16.731335 22501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [term 1 LEADER]: Becoming Leader. State: Replica: 1cce8029d33641c296e7c92ae5be7448, State: Running, Role: LEADER
I20260809 00:53:16.732156 22501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cce8029d33641c296e7c92ae5be7448" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41961 } }
I20260809 00:53:16.732990 22487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:53:16.735555 22504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cce8029d33641c296e7c92ae5be7448. Latest consensus state: current_term: 1 leader_uuid: "1cce8029d33641c296e7c92ae5be7448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cce8029d33641c296e7c92ae5be7448" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41961 } } }
I20260809 00:53:16.736073 22504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:16.736817 22502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cce8029d33641c296e7c92ae5be7448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cce8029d33641c296e7c92ae5be7448" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41961 } } }
I20260809 00:53:16.737138 22502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448 [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:16.737226 22512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:53:16.744652 22512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:53:16.755585 22512 catalog_manager.cc:1347] Generated new cluster ID: 7ddec4b8af0c4e52a16b21f6ab2f1a0a
I20260809 00:53:16.755738 22512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:53:16.773696 22512 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:53:16.775276 22512 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:53:16.786479 22512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448: Generated new TSK 0
I20260809 00:53:16.787495 22512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab.
Entry for principal oryx/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab.
I20260809 00:53:16.875689 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:41961
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab
--principal=oryx/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:17.293951 22528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:17.294327 22528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:17.294503 22528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:17.302572 22528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:17.302778 22528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:17.302927 22528 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 00:53:17.303076 22528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:53:17.313987 22528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab
--principal=oryx/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.225.62:41961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:17.315728 22528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:17.317512 22528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:17.332922 22538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:17.333334 22534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:17.333077 22541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:17.338814 22528 server_base.cc:1034] running on GCE node
I20260809 00:53:17.339679 22528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:17.340834 22528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:17.342118 22528 hybrid_clock.cc:648] HybridClock initialized: now 1786236797342038 us; error 57 us; skew 500 ppm
I20260809 00:53:17.342491 22528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:17 dist-test-slave-j2vl krb5kdc[22341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236797, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:17.346208 22528 init.cc:377] Logged in from keytab as oryx/127.9.225.1@KRBTEST.COM (short username oryx)
I20260809 00:53:17.348002 22528 webserver.cc:460] Webserver started at http://127.9.225.1:33145/ using document root <none> and password file <none>
I20260809 00:53:17.348865 22528 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:17.349020 22528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:17.349472 22528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:17.352818 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "92771243eee44cef9d4d1f7626f58947"
format_stamp: "Formatted at 2026-08-09 00:53:17 on dist-test-slave-j2vl"
server_key: "b0b3de4724c663dcc5cd1a7d69c4575d"
server_key_iv: "b8697622aac3f9bfef61d5174c5596dc"
server_key_version: "encryptionkey@0"
I20260809 00:53:17.353792 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "92771243eee44cef9d4d1f7626f58947"
format_stamp: "Formatted at 2026-08-09 00:53:17 on dist-test-slave-j2vl"
server_key: "b0b3de4724c663dcc5cd1a7d69c4575d"
server_key_iv: "b8697622aac3f9bfef61d5174c5596dc"
server_key_version: "encryptionkey@0"
I20260809 00:53:17.359877 22528 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:53:17.364454 22552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:17.367306 22528 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260809 00:53:17.367578 22528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "92771243eee44cef9d4d1f7626f58947"
format_stamp: "Formatted at 2026-08-09 00:53:17 on dist-test-slave-j2vl"
server_key: "b0b3de4724c663dcc5cd1a7d69c4575d"
server_key_iv: "b8697622aac3f9bfef61d5174c5596dc"
server_key_version: "encryptionkey@0"
I20260809 00:53:17.367871 22528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:17.386204 22528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:17.391263 22528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:17.391613 22528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:17.395423 22528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:17.395565 22528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:17.395828 22528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:17.395918 22528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 00:53:17 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236797, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:17.424365 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:17.396010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:40851 (local address 127.9.225.62:41961)
0809 00:53:17.397007 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:17.397015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:17.398206 (+ 1191us) server_negotiation.cc:407] Connection header received
0809 00:53:17.400537 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:17.400545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:17.400670 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:17.400861 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:17.403810 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:17.405157 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:17.407981 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:17.408786 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:17.408989 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:17.414777 (+ 5788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:17.414784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:17.414787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:17.414834 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:17.418779 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:17.420399 (+ 1620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:17.420404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:17.420406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:17.420553 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:17.421301 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:17.421306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:17.421308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:17.421883 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:17.422081 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:17.423392 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0809 00:53:17.423767 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":671,"threads_started":1}
I20260809 00:53:17.424906 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:17.396906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:41961 (local address 127.9.225.1:40851)
0809 00:53:17.397964 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:17.398031 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:17.400165 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:17.401384 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:17.401406 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:17.402365 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:17.403453 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:17.403480 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:17.405442 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:17.405452 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:17.407753 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:17.407770 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:17.409117 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:17.409125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:17.409598 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:17.410638 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:17.410681 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:17.414351 (+ 3670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:17.419019 (+ 4668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:17.419034 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:17.419063 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:17.420163 (+ 1100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:17.420695 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:17.420703 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:17.420710 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:17.421118 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:17.422063 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:17.422073 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:17.422714 (+ 641us) client_negotiation.cc:769] Sending connection context
0809 00:53:17.423157 (+ 443us) client_negotiation.cc:240] Negotiation successful
0809 00:53:17.423685 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":442,"threads_started":1}
I20260809 00:53:17.449461 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:17.432255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:55823 (local address 127.9.225.62:41961)
0809 00:53:17.432424 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:17.432428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:17.433173 (+ 745us) server_negotiation.cc:407] Connection header received
0809 00:53:17.433422 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:17.433432 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:17.433506 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:17.433622 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:17.435270 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:17.436273 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:17.438761 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:17.439269 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:17.439358 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:17.442442 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:17.442456 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:17.442460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:17.442498 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:17.446346 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:17.447147 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:17.447160 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:17.447162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:17.447301 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:17.447711 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:17.447716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:17.447719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:17.448034 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:17.448293 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:17.448759 (+ 466us) server_negotiation.cc:299] Negotiation successful
0809 00:53:17.449012 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 00:53:17.449560 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:17.432271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:41961 (local address 127.9.225.1:55823)
0809 00:53:17.433049 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:17.433091 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:17.433250 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:17.433796 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:17.433844 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:17.434491 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:17.435025 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:17.435047 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:17.436594 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:17.436606 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:17.438513 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:17.438542 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:17.439642 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:17.439647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:17.439782 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:17.440603 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:17.440638 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:17.442113 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:17.446526 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:17.446532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:17.446537 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:17.446997 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:17.447439 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:17.447444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:17.447448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:17.447587 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:17.448162 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:17.448166 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:17.448418 (+ 252us) client_negotiation.cc:769] Sending connection context
0809 00:53:17.448578 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 00:53:17.449001 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":479,"threads_started":1}
I20260809 00:53:17.452701 22748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:40685 every 8 connection(s)
I20260809 00:53:17.452680 22528 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:40685
I20260809 00:53:17.454886 22528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:53:17.457440 22749 heartbeater.cc:347] Connected to a master server at 127.9.225.62:41961
I20260809 00:53:17.457677 22749 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:17.458400 22749 heartbeater.cc:511] Master 127.9.225.62:41961 requested a full tablet report, sending...
I20260809 00:53:17.460839 22422 ts_manager.cc:194] Registered new tserver with Master: 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685)
I20260809 00:53:17.462703 22422 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:40851
I20260809 00:53:17.462869 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 22528
I20260809 00:53:17.463116 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:53:17.463927 10116 external_mini_cluster.cc:1428] Setting key 9a99f46d0eec49f6efe7305743ee7d77
WARNING: no policy specified for oryx/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab.
Entry for principal oryx/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab.
I20260809 00:53:17.529325 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:41961
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab
--principal=oryx/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:17.916471 22755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:17.916826 22755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:17.916932 22755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:17.924676 22755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:17.924830 22755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:17.924922 22755 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 00:53:17.925009 22755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:53:17.936204 22755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab
--principal=oryx/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.225.62:41961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:17.937748 22755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:17.939159 22755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:17.955622 22764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:17.955647 22765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:17.955835 22769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:17.956598 22755 server_base.cc:1034] running on GCE node
I20260809 00:53:17.957320 22755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:17.958500 22755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:17.959774 22755 hybrid_clock.cc:648] HybridClock initialized: now 1786236797959689 us; error 49 us; skew 500 ppm
I20260809 00:53:17.960160 22755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:17 dist-test-slave-j2vl krb5kdc[22341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236797, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:17.963864 22755 init.cc:377] Logged in from keytab as oryx/127.9.225.2@KRBTEST.COM (short username oryx)
I20260809 00:53:17.965724 22755 webserver.cc:460] Webserver started at http://127.9.225.2:33349/ using document root <none> and password file <none>
I20260809 00:53:17.966580 22755 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:17.966689 22755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:17.967100 22755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:17.970968 22755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "686eb4031a3f46e6be453b4799ca13ea"
format_stamp: "Formatted at 2026-08-09 00:53:17 on dist-test-slave-j2vl"
server_key: "48e78c8c5017f90f0d54eff14e0dd96f"
server_key_iv: "679539d9b3600b9aac223ba3224a1d15"
server_key_version: "encryptionkey@0"
I20260809 00:53:17.972028 22755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "686eb4031a3f46e6be453b4799ca13ea"
format_stamp: "Formatted at 2026-08-09 00:53:17 on dist-test-slave-j2vl"
server_key: "48e78c8c5017f90f0d54eff14e0dd96f"
server_key_iv: "679539d9b3600b9aac223ba3224a1d15"
server_key_version: "encryptionkey@0"
I20260809 00:53:17.978318 22755 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 00:53:17.982313 22779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:17.983932 22755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 00:53:17.984162 22755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "686eb4031a3f46e6be453b4799ca13ea"
format_stamp: "Formatted at 2026-08-09 00:53:17 on dist-test-slave-j2vl"
server_key: "48e78c8c5017f90f0d54eff14e0dd96f"
server_key_iv: "679539d9b3600b9aac223ba3224a1d15"
server_key_version: "encryptionkey@0"
I20260809 00:53:17.984443 22755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:17.998041 22755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:18.003280 22755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:18.003729 22755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:18.008261 22755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:18.008397 22755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:18.008639 22755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:18.008733 22755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 00:53:18 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236797, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:18.040076 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.009704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:40393 (local address 127.9.225.62:41961)
0809 00:53:18.010317 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:18.010327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:18.011275 (+ 948us) server_negotiation.cc:407] Connection header received
0809 00:53:18.014610 (+ 3335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:18.014618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:18.014748 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:18.014832 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:18.019123 (+ 4291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.020251 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.023010 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.023700 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.023869 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.029804 (+ 5935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:18.029816 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:18.029824 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:18.029892 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:18.034586 (+ 4694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:18.036132 (+ 1546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:18.036141 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:18.036147 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:18.036361 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:18.037103 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:18.037111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:18.037117 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:18.037631 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:18.037845 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:18.039138 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0809 00:53:18.039483 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":310,"threads_started":1}
I20260809 00:53:18.042259 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.009932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:41961 (local address 127.9.225.2:40393)
0809 00:53:18.011053 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:18.011117 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:18.014189 (+ 3072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:18.015425 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:18.015438 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:18.017499 (+ 2061us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:18.018799 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.018841 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.020551 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.020561 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.022781 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.022803 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.024008 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.024016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.024518 (+ 502us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.025569 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:18.025600 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:18.029282 (+ 3682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:18.034930 (+ 5648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:18.034945 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:18.034981 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:18.035952 (+ 971us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:18.036493 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:18.036500 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:18.036506 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:18.036873 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:18.037831 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:18.037844 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:18.038537 (+ 693us) client_negotiation.cc:769] Sending connection context
0809 00:53:18.038967 (+ 430us) client_negotiation.cc:240] Negotiation successful
0809 00:53:18.039754 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":456,"threads_started":1}
I20260809 00:53:18.061020 22755 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:41509
I20260809 00:53:18.061084 22979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:41509 every 8 connection(s)
I20260809 00:53:18.063257 22755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:53:18.066821 22980 heartbeater.cc:347] Connected to a master server at 127.9.225.62:41961
I20260809 00:53:18.067130 22980 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:18.068171 22980 heartbeater.cc:511] Master 127.9.225.62:41961 requested a full tablet report, sending...
I20260809 00:53:18.070595 22422 ts_manager.cc:194] Registered new tserver with Master: 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509)
I20260809 00:53:18.071774 22422 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:40393
I20260809 00:53:18.072665 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 22755
I20260809 00:53:18.072788 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.052782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:41961 (local address 127.9.225.2:46219)
0809 00:53:18.054364 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:18.054391 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:18.054769 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:18.055470 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:18.055476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:18.056255 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:18.056571 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.056582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.058304 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.058315 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.060223 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.060250 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.061562 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.061569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.061757 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.062513 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:18.062545 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:18.064227 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:18.069675 (+ 5448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:18.069681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:18.069685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:18.070250 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:18.070741 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:18.070746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:18.070749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:18.070922 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:18.071587 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:18.071592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:18.071812 (+ 220us) client_negotiation.cc:769] Sending connection context
0809 00:53:18.071987 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 00:53:18.072357 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1338,"thread_start_us":1129,"threads_started":1}
I20260809 00:53:18.072840 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:53:18.072906 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.052832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:46219 (local address 127.9.225.62:41961)
0809 00:53:18.053125 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:18.053136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:18.054536 (+ 1400us) server_negotiation.cc:407] Connection header received
0809 00:53:18.054985 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:18.054992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:18.055094 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:18.055184 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:18.056746 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.058054 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.060495 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.061251 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.061499 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.064694 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:18.064702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:18.064709 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:18.064762 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:18.069335 (+ 4573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:18.070396 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:18.070401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:18.070403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:18.070583 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:18.071067 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:18.071071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:18.071074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:18.071448 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:18.071567 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:18.072211 (+ 644us) server_negotiation.cc:299] Negotiation successful
0809 00:53:18.072527 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 00:53:18.073632 10116 external_mini_cluster.cc:1428] Setting key 62cda6a67a3dd325277ec5db6427f345
WARNING: no policy specified for oryx/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab.
Entry for principal oryx/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab.
I20260809 00:53:18.142352 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:41961
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab
--principal=oryx/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:53:18.466810 22749 heartbeater.cc:503] Master 127.9.225.62:41961 was elected leader, sending a full tablet report...
W20260809 00:53:18.529600 22985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:18.529929 22985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:18.530100 22985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:18.538177 22985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:18.538321 22985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:18.538471 22985 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 00:53:18.538568 22985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:53:18.549407 22985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab
--principal=oryx/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.225.62:41961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:18.550961 22985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:18.552598 22985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:18.567960 22992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:18.567952 22991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:18.568001 22994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:18.569471 22985 server_base.cc:1034] running on GCE node
I20260809 00:53:18.570254 22985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:18.571439 22985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:18.572796 22985 hybrid_clock.cc:648] HybridClock initialized: now 1786236798572655 us; error 55 us; skew 500 ppm
I20260809 00:53:18.573299 22985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:18 dist-test-slave-j2vl krb5kdc[22341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236798, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:18.580586 22985 init.cc:377] Logged in from keytab as oryx/127.9.225.3@KRBTEST.COM (short username oryx)
I20260809 00:53:18.582440 22985 webserver.cc:460] Webserver started at http://127.9.225.3:36515/ using document root <none> and password file <none>
I20260809 00:53:18.583328 22985 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:18.583433 22985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:18.583741 22985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:18.587155 22985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "6d57d79c649a4a1fb38bb4a9c22ba752"
format_stamp: "Formatted at 2026-08-09 00:53:18 on dist-test-slave-j2vl"
server_key: "4bcf6a9e98ffc916069655cd82d047ae"
server_key_iv: "ee228adbf41c9ea0e8967b410626a4bc"
server_key_version: "encryptionkey@0"
I20260809 00:53:18.588127 22985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "6d57d79c649a4a1fb38bb4a9c22ba752"
format_stamp: "Formatted at 2026-08-09 00:53:18 on dist-test-slave-j2vl"
server_key: "4bcf6a9e98ffc916069655cd82d047ae"
server_key_iv: "ee228adbf41c9ea0e8967b410626a4bc"
server_key_version: "encryptionkey@0"
I20260809 00:53:18.594430 22985 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 00:53:18.598765 23006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:18.600469 22985 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260809 00:53:18.600668 22985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "6d57d79c649a4a1fb38bb4a9c22ba752"
format_stamp: "Formatted at 2026-08-09 00:53:18 on dist-test-slave-j2vl"
server_key: "4bcf6a9e98ffc916069655cd82d047ae"
server_key_iv: "ee228adbf41c9ea0e8967b410626a4bc"
server_key_version: "encryptionkey@0"
I20260809 00:53:18.600919 22985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:18.622721 22985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:18.627929 22985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:18.628294 22985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:18.631562 22985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:18.631685 22985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:18.631870 22985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:18.631949 22985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 00:53:18 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236798, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:18.658324 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.632473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:38025 (local address 127.9.225.62:41961)
0809 00:53:18.635704 (+ 3231us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:18.635708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:18.635728 (+ 20us) server_negotiation.cc:407] Connection header received
0809 00:53:18.638769 (+ 3041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:18.638776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:18.638892 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:18.639035 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:18.641659 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.642435 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.644165 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.644617 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.644764 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.650428 (+ 5664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:18.650438 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:18.650445 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:18.650504 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:18.654506 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:18.655459 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:18.655465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:18.655467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:18.655608 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:18.656127 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:18.656131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:18.656133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:18.656497 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:18.657494 (+ 997us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:18.657637 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 00:53:18.657913 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3116,"thread_start_us":2971,"threads_started":1}
I20260809 00:53:18.659948 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.633389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:41961 (local address 127.9.225.3:38025)
0809 00:53:18.634743 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:18.634813 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:18.638218 (+ 3405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:18.639335 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:18.639348 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:18.640244 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:18.641425 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.641454 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.642627 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.642633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.644003 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.644016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.644912 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.644922 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.645158 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.646034 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:18.646067 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:18.650012 (+ 3945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:18.654707 (+ 4695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:18.654733 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:18.654764 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:18.655293 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:18.655739 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:18.655744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:18.655748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:18.655989 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:18.656672 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:18.656682 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:18.657132 (+ 450us) client_negotiation.cc:769] Sending connection context
0809 00:53:18.657454 (+ 322us) client_negotiation.cc:240] Negotiation successful
0809 00:53:18.658116 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"thread_start_us":488,"threads_started":1}
I20260809 00:53:18.689175 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.669228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:41961 (local address 127.9.225.3:37341)
0809 00:53:18.670414 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:18.670455 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:18.670628 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:18.671090 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:18.671093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:18.671629 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:18.672065 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.672078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.674059 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.674067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.675388 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.675406 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.676771 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.676776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.676924 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.677789 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:18.677829 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:18.679637 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:18.685315 (+ 5678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:18.685321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:18.685326 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:18.685979 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:18.686644 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:18.686649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:18.686653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:18.686874 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:18.687775 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:18.687781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:18.687998 (+ 217us) client_negotiation.cc:769] Sending connection context
0809 00:53:18.688164 (+ 166us) client_negotiation.cc:240] Negotiation successful
0809 00:53:18.688679 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":743,"threads_started":1}
I20260809 00:53:18.689301 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.669211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:37341 (local address 127.9.225.62:41961)
0809 00:53:18.669546 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:18.669555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:18.670578 (+ 1023us) server_negotiation.cc:407] Connection header received
0809 00:53:18.670773 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:18.670782 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:18.670919 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:18.671069 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:18.672246 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.673835 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.675590 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.676547 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.676735 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.679988 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:18.679997 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:18.680004 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:18.680062 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:18.685109 (+ 5047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:18.686185 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:18.686196 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:18.686202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:18.686438 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:18.687058 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:18.687065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:18.687068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:18.687659 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:18.687816 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:18.688403 (+ 587us) server_negotiation.cc:299] Negotiation successful
0809 00:53:18.688800 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260809 00:53:18.700881 22985 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:36925
I20260809 00:53:18.701171 23222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:36925 every 8 connection(s)
I20260809 00:53:18.702682 22985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:53:18.705549 23224 heartbeater.cc:347] Connected to a master server at 127.9.225.62:41961
I20260809 00:53:18.705648 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 22985
I20260809 00:53:18.705848 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:53:18.705907 23224 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:18.706465 10116 external_mini_cluster.cc:1428] Setting key 61e540b4b2d5e33c2cbc7fe7a8fa6d84
I20260809 00:53:18.706915 23224 heartbeater.cc:511] Master 127.9.225.62:41961 requested a full tablet report, sending...
I20260809 00:53:18.708778 22422 ts_manager.cc:194] Registered new tserver with Master: 6d57d79c649a4a1fb38bb4a9c22ba752 (127.9.225.3:36925)
I20260809 00:53:18.709811 22422 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:38025
I20260809 00:53:18.710343 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 00:53:18 dist-test-slave-j2vl krb5kdc[22341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236798, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 00:53:18 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:18 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 00:53:18.780802 23021 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:18.768144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.9.225.62:41961)
0809 00:53:18.768359 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:18.768364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:18.768749 (+ 385us) server_negotiation.cc:407] Connection header received
0809 00:53:18.769140 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:18.769164 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:18.769333 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:18.769534 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:18.771237 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.772040 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.773606 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.774073 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.774186 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.780550 (+ 6364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34318: BlockingRecv error: recv got EOF from 127.0.0.1:34318 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 09 00:53:18 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236798, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:18.801174 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.784897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.9.225.62:41961)
0809 00:53:18.785181 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:18.785190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:18.785567 (+ 377us) server_negotiation.cc:407] Connection header received
0809 00:53:18.785828 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:18.785833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:18.785901 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:18.786013 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:18.787289 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.788035 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.789501 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.789991 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.790143 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.794527 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:18.794534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:18.794538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:18.794584 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:18.797915 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:18.798834 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:18.798839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:18.798842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:18.799015 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:18.799465 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:18.799470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:18.799472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:18.799984 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:18.800246 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:18.800561 (+ 315us) server_negotiation.cc:299] Negotiation successful
0809 00:53:18.800778 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 00:53:18.806556 22422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:53:18.814188 22422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:53:18.847892 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.829306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:36925 (local address 127.0.0.1:40828)
0809 00:53:18.832526 (+ 3220us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:18.832553 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:18.832718 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:18.834245 (+ 1527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:18.834257 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:18.834306 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:18.834908 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.834927 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.837970 (+ 3043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.837980 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.841649 (+ 3669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.841688 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.846452 (+ 4764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.846463 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.846878 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.846977 (+ 99us) client_negotiation.cc:769] Sending connection context
0809 00:53:18.847188 (+ 211us) client_negotiation.cc:240] Negotiation successful
0809 00:53:18.847480 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3061,"thread_start_us":466,"threads_started":1}
I20260809 00:53:18.848337 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.829952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.9.225.3:36925)
0809 00:53:18.832269 (+ 2317us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:18.832280 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:18.832886 (+ 606us) server_negotiation.cc:407] Connection header received
0809 00:53:18.833035 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:18.833048 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:18.833479 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:18.833737 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:18.835220 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.837677 (+ 2457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.842042 (+ 4365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.844379 (+ 2337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.844589 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.844963 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:18.847358 (+ 2395us) server_negotiation.cc:299] Negotiation successful
0809 00:53:18.847659 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2116,"thread_start_us":1935,"threads_started":1}
I20260809 00:53:18.852705 23099 tablet_service.cc:1467] Processing CreateTablet for tablet b5b24931cfed4a678af3d27a1b23f841 (DEFAULT_TABLE table=test-table [id=c5f23872b2354abfa845c14cf3ad87c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:18.854658 23099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5b24931cfed4a678af3d27a1b23f841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:18.857208 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.832434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.9.225.1:40685)
0809 00:53:18.847829 (+ 15395us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:18.847839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:18.847877 (+ 38us) server_negotiation.cc:407] Connection header received
0809 00:53:18.848010 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:18.848019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:18.848394 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:18.848615 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:18.849535 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.851188 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.853774 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.855590 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.855765 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.855996 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:18.856687 (+ 691us) server_negotiation.cc:299] Negotiation successful
0809 00:53:18.856897 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15202,"thread_start_us":16008,"threads_started":1}
I20260809 00:53:18.857388 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.828982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:41509 (local address 127.0.0.1:32960)
0809 00:53:18.830348 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:18.830401 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:18.830605 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:18.849672 (+ 19067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:18.849677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:18.849709 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:18.850082 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.850096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.852476 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.852482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.854432 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.854454 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.856428 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.856435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.856609 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.856652 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 00:53:18.856754 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 00:53:18.856932 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"thread_start_us":2525,"threads_started":1}
I20260809 00:53:18.857107 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.831648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:40685 (local address 127.0.0.1:43494)
0809 00:53:18.845751 (+ 14103us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:18.845789 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:18.846114 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:18.848753 (+ 2639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:18.848758 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:18.848791 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:18.849233 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.849248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.851382 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.851388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.853540 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:18.853561 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.856148 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.856155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:18.856364 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.856415 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 00:53:18.856503 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 00:53:18.856678 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13861,"thread_start_us":13506,"threads_started":1}
I20260809 00:53:18.857743 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:18.829443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.9.225.2:41509)
0809 00:53:18.848047 (+ 18604us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:18.848059 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:18.848100 (+ 41us) server_negotiation.cc:407] Connection header received
0809 00:53:18.849107 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:18.849120 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:18.849429 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:18.850314 (+ 885us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:18.850556 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.852137 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.854712 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:18.856171 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:18.856329 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:18.856545 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:18.857219 (+ 674us) server_negotiation.cc:299] Negotiation successful
0809 00:53:18.857442 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18390,"thread_start_us":19345,"threads_started":1}
I20260809 00:53:18.862032 22629 tablet_service.cc:1467] Processing CreateTablet for tablet b5b24931cfed4a678af3d27a1b23f841 (DEFAULT_TABLE table=test-table [id=c5f23872b2354abfa845c14cf3ad87c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:18.862037 22859 tablet_service.cc:1467] Processing CreateTablet for tablet b5b24931cfed4a678af3d27a1b23f841 (DEFAULT_TABLE table=test-table [id=c5f23872b2354abfa845c14cf3ad87c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:18.863977 22629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5b24931cfed4a678af3d27a1b23f841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:18.864003 22859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5b24931cfed4a678af3d27a1b23f841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:18.874567 23260 tablet_bootstrap.cc:492] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752: Bootstrap starting.
I20260809 00:53:18.878973 23260 tablet_bootstrap.cc:654] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:18.880839 23260 log.cc:824] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:18.881219 23261 tablet_bootstrap.cc:492] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea: Bootstrap starting.
I20260809 00:53:18.883436 23262 tablet_bootstrap.cc:492] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947: Bootstrap starting.
I20260809 00:53:18.886740 23261 tablet_bootstrap.cc:654] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:18.887611 23262 tablet_bootstrap.cc:654] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:18.888357 23261 log.cc:824] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:18.890026 23262 log.cc:824] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:18.891496 23260 tablet_bootstrap.cc:492] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752: No bootstrap required, opened a new log
I20260809 00:53:18.891830 23260 ts_tablet_manager.cc:1397] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.009s
I20260809 00:53:18.892001 23261 tablet_bootstrap.cc:492] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea: No bootstrap required, opened a new log
I20260809 00:53:18.892310 23261 ts_tablet_manager.cc:1397] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260809 00:53:18.893503 23262 tablet_bootstrap.cc:492] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947: No bootstrap required, opened a new log
I20260809 00:53:18.893802 23262 ts_tablet_manager.cc:1397] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260809 00:53:18.900835 23260 raft_consensus.cc:348] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } }
I20260809 00:53:18.901438 23260 raft_consensus.cc:374] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:18.901554 23260 raft_consensus.cc:729] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d57d79c649a4a1fb38bb4a9c22ba752, State: Initialized, Role: FOLLOWER
I20260809 00:53:18.902359 23260 consensus_queue.cc:260] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } }
I20260809 00:53:18.903684 23224 heartbeater.cc:503] Master 127.9.225.62:41961 was elected leader, sending a full tablet report...
I20260809 00:53:18.904527 23260 ts_tablet_manager.cc:1428] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260809 00:53:18.904541 23261 raft_consensus.cc:348] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } }
I20260809 00:53:18.905102 23261 raft_consensus.cc:374] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:18.905216 23261 raft_consensus.cc:729] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 686eb4031a3f46e6be453b4799ca13ea, State: Initialized, Role: FOLLOWER
I20260809 00:53:18.905987 23261 consensus_queue.cc:260] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } }
I20260809 00:53:18.907253 22980 heartbeater.cc:503] Master 127.9.225.62:41961 was elected leader, sending a full tablet report...
I20260809 00:53:18.907999 23261 ts_tablet_manager.cc:1428] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260809 00:53:18.906806 23262 raft_consensus.cc:348] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } }
I20260809 00:53:18.908915 23262 raft_consensus.cc:374] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:18.909039 23262 raft_consensus.cc:729] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92771243eee44cef9d4d1f7626f58947, State: Initialized, Role: FOLLOWER
I20260809 00:53:18.909848 23262 consensus_queue.cc:260] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } }
I20260809 00:53:18.911957 23262 ts_tablet_manager.cc:1428] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947: Time spent starting tablet: real 0.018s user 0.007s sys 0.008s
W20260809 00:53:18.954799 23225 tablet.cc:2369] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:53:18.957970 22751 tablet.cc:2369] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:53:19.008605 23271 raft_consensus.cc:482] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:53:19.008971 23271 raft_consensus.cc:504] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } }
I20260809 00:53:19.010932 23271 leader_election.cc:288] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d57d79c649a4a1fb38bb4a9c22ba752 (127.9.225.3:36925), 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685)
I20260809 00:53:19.018644 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.011530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:55301 (local address 127.9.225.3:36925)
0809 00:53:19.011836 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:19.011843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:19.012089 (+ 246us) server_negotiation.cc:407] Connection header received
0809 00:53:19.012336 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:19.012344 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:19.012443 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:19.012549 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:19.013237 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.014308 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.016407 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.017577 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.017683 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.017776 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:19.018200 (+ 424us) server_negotiation.cc:299] Negotiation successful
0809 00:53:19.018309 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 00:53:19.018762 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.011226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:36925 (local address 127.9.225.2:55301)
0809 00:53:19.011976 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:19.012000 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:19.012183 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:19.012597 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:19.012601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:19.012643 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:19.013066 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.013082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.014459 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.014464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.016217 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.016231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.017809 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.017817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.018012 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.018054 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 00:53:19.018164 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 00:53:19.018322 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":324,"threads_started":1}
I20260809 00:53:19.019411 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.012205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:56801 (local address 127.9.225.1:40685)
0809 00:53:19.012483 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:19.012491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:19.012977 (+ 486us) server_negotiation.cc:407] Connection header received
0809 00:53:19.013132 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:19.013140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:19.013254 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:19.013325 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:19.014057 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.015075 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.017109 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.018274 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.018407 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.018526 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:19.018947 (+ 421us) server_negotiation.cc:299] Negotiation successful
0809 00:53:19.019123 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 00:53:19.019987 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.011979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:40685 (local address 127.9.225.2:56801)
0809 00:53:19.012835 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:19.012851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:19.012962 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:19.013415 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:19.013419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:19.013446 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:19.013863 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.013876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.015243 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.015246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.016928 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.016943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.018524 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.018530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.018734 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.018768 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 00:53:19.019567 (+ 799us) client_negotiation.cc:240] Negotiation successful
0809 00:53:19.019734 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":492,"threads_started":1}
I20260809 00:53:19.020192 23141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5b24931cfed4a678af3d27a1b23f841" candidate_uuid: "686eb4031a3f46e6be453b4799ca13ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" is_pre_election: true
I20260809 00:53:19.020813 23141 raft_consensus.cc:2454] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 686eb4031a3f46e6be453b4799ca13ea in term 0.
I20260809 00:53:19.020918 22661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5b24931cfed4a678af3d27a1b23f841" candidate_uuid: "686eb4031a3f46e6be453b4799ca13ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92771243eee44cef9d4d1f7626f58947" is_pre_election: true
I20260809 00:53:19.021548 22661 raft_consensus.cc:2454] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 686eb4031a3f46e6be453b4799ca13ea in term 0.
I20260809 00:53:19.021680 22784 leader_election.cc:302] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 686eb4031a3f46e6be453b4799ca13ea, 6d57d79c649a4a1fb38bb4a9c22ba752; no voters:
I20260809 00:53:19.022194 23271 raft_consensus.cc:2781] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:53:19.022374 23271 raft_consensus.cc:482] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:53:19.022506 23271 raft_consensus.cc:3037] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:19.024946 23271 raft_consensus.cc:504] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } }
I20260809 00:53:19.025712 23271 leader_election.cc:288] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [CANDIDATE]: Term 1 election: Requested vote from peers 6d57d79c649a4a1fb38bb4a9c22ba752 (127.9.225.3:36925), 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685)
I20260809 00:53:19.026542 23141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5b24931cfed4a678af3d27a1b23f841" candidate_uuid: "686eb4031a3f46e6be453b4799ca13ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752"
I20260809 00:53:19.026542 22661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5b24931cfed4a678af3d27a1b23f841" candidate_uuid: "686eb4031a3f46e6be453b4799ca13ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92771243eee44cef9d4d1f7626f58947"
I20260809 00:53:19.026810 23141 raft_consensus.cc:3037] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:19.026810 22661 raft_consensus.cc:3037] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:19.030411 22661 raft_consensus.cc:2454] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 686eb4031a3f46e6be453b4799ca13ea in term 1.
I20260809 00:53:19.030442 23141 raft_consensus.cc:2454] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 686eb4031a3f46e6be453b4799ca13ea in term 1.
I20260809 00:53:19.031000 22784 leader_election.cc:302] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 686eb4031a3f46e6be453b4799ca13ea, 92771243eee44cef9d4d1f7626f58947; no voters:
I20260809 00:53:19.031414 23271 raft_consensus.cc:2781] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:19.031975 23271 raft_consensus.cc:686] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 1 LEADER]: Becoming Leader. State: Replica: 686eb4031a3f46e6be453b4799ca13ea, State: Running, Role: LEADER
I20260809 00:53:19.032663 23271 consensus_queue.cc:237] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } }
I20260809 00:53:19.038733 22420 catalog_manager.cc:5495] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2). New cstate: current_term: 1 leader_uuid: "686eb4031a3f46e6be453b4799ca13ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } health_report { overall_health: UNKNOWN } } }
W20260809 00:53:19.067263 22982 tablet.cc:2369] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:53:19.087738 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.081832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.9.225.2:41509)
0809 00:53:19.082092 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:19.082099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:19.082120 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:53:19.082381 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:19.082405 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:19.082534 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:19.082708 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:53:19.083090 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.083846 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.085219 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.085727 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.085878 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.086387 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:53:19.087060 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:53:19.087231 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:19.087327 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 00:53:19.087427 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 00:53:19.097635 22661 raft_consensus.cc:1261] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Refusing update from remote peer 686eb4031a3f46e6be453b4799ca13ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:19.097635 23141 raft_consensus.cc:1261] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Refusing update from remote peer 686eb4031a3f46e6be453b4799ca13ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:19.098933 23278 consensus_queue.cc:1035] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:19.099378 23271 consensus_queue.cc:1035] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:19.114492 23282 mvcc.cc:205] Tried to move back new op lower bound from 7316425929090932736 to 7316425928843640832. Current Snapshot: MvccSnapshot[applied={T|T < 7316425929090932736}]
I20260809 00:53:19.114487 23283 mvcc.cc:205] Tried to move back new op lower bound from 7316425929090932736 to 7316425928843640832. Current Snapshot: MvccSnapshot[applied={T|T < 7316425929090932736}]
I20260809 00:53:19.129917 22420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:19.130234 22420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:19.134793 22420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448: Sending DeleteTablet for 3 replicas of tablet b5b24931cfed4a678af3d27a1b23f841
I20260809 00:53:19.136433 22859 tablet_service.cc:1514] Processing DeleteTablet for tablet b5b24931cfed4a678af3d27a1b23f841 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:19 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:32960
I20260809 00:53:19.136802 23099 tablet_service.cc:1514] Processing DeleteTablet for tablet b5b24931cfed4a678af3d27a1b23f841 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:19 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:40828
I20260809 00:53:19.137744 22629 tablet_service.cc:1514] Processing DeleteTablet for tablet b5b24931cfed4a678af3d27a1b23f841 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:19 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:43494
I20260809 00:53:19.138058 22422 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:53:19.138959 22422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:53:19.142501 23297 tablet_replica.cc:330] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752: stopping tablet replica
I20260809 00:53:19.143064 23297 raft_consensus.cc:2229] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:19.143671 23297 raft_consensus.cc:2258] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:19.146353 23297 ts_tablet_manager.cc:1905] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:19.154094 22857 tablet_service.cc:1467] Processing CreateTablet for tablet c74e0b4752e54fe9b08ae37bfef1a5bb (DEFAULT_TABLE table=test-table [id=d9e389a01d2042b7b01c0725625e3007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:19.154546 23297 ts_tablet_manager.cc:1918] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:19.154794 23297 log.cc:1196] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/b5b24931cfed4a678af3d27a1b23f841
I20260809 00:53:19.154919 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74e0b4752e54fe9b08ae37bfef1a5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:19.154770 23097 tablet_service.cc:1467] Processing CreateTablet for tablet c74e0b4752e54fe9b08ae37bfef1a5bb (DEFAULT_TABLE table=test-table [id=d9e389a01d2042b7b01c0725625e3007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:19.155351 23097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74e0b4752e54fe9b08ae37bfef1a5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:19.155534 23297 ts_tablet_manager.cc:1939] T b5b24931cfed4a678af3d27a1b23f841 P 6d57d79c649a4a1fb38bb4a9c22ba752: Deleting consensus metadata
I20260809 00:53:19.157850 22401 catalog_manager.cc:4841] TS 6d57d79c649a4a1fb38bb4a9c22ba752 (127.9.225.3:36925): tablet b5b24931cfed4a678af3d27a1b23f841 (table test-table [id=c5f23872b2354abfa845c14cf3ad87c1]) successfully deleted
I20260809 00:53:19.158284 22628 tablet_service.cc:1467] Processing CreateTablet for tablet c74e0b4752e54fe9b08ae37bfef1a5bb (DEFAULT_TABLE table=test-table [id=d9e389a01d2042b7b01c0725625e3007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:19.158958 22628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74e0b4752e54fe9b08ae37bfef1a5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:19.162398 23261 tablet_bootstrap.cc:492] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea: Bootstrap starting.
I20260809 00:53:19.164317 23261 tablet_bootstrap.cc:654] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:19.165139 23262 tablet_bootstrap.cc:492] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947: Bootstrap starting.
I20260809 00:53:19.167438 23262 tablet_bootstrap.cc:654] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:19.175228 23261 tablet_bootstrap.cc:492] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea: No bootstrap required, opened a new log
I20260809 00:53:19.175288 23260 tablet_bootstrap.cc:492] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752: Bootstrap starting.
I20260809 00:53:19.175590 23261 ts_tablet_manager.cc:1397] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20260809 00:53:19.176715 23261 raft_consensus.cc:348] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:19.177204 23261 raft_consensus.cc:374] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:19.177505 23261 raft_consensus.cc:729] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 686eb4031a3f46e6be453b4799ca13ea, State: Initialized, Role: FOLLOWER
I20260809 00:53:19.178027 23261 consensus_queue.cc:260] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:19.178445 23260 tablet_bootstrap.cc:654] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:19.179088 23261 ts_tablet_manager.cc:1428] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 00:53:19.181861 23262 tablet_bootstrap.cc:492] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947: No bootstrap required, opened a new log
I20260809 00:53:19.182067 23262 ts_tablet_manager.cc:1397] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.001s
I20260809 00:53:19.182686 23260 tablet_bootstrap.cc:492] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752: No bootstrap required, opened a new log
I20260809 00:53:19.182870 23260 ts_tablet_manager.cc:1397] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20260809 00:53:19.183036 23262 raft_consensus.cc:348] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:19.183313 23262 raft_consensus.cc:374] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:19.183427 23262 raft_consensus.cc:729] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92771243eee44cef9d4d1f7626f58947, State: Initialized, Role: FOLLOWER
I20260809 00:53:19.183794 23262 consensus_queue.cc:260] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:19.184157 23260 raft_consensus.cc:348] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:19.184445 23260 raft_consensus.cc:374] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:19.184558 23260 raft_consensus.cc:729] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d57d79c649a4a1fb38bb4a9c22ba752, State: Initialized, Role: FOLLOWER
I20260809 00:53:19.184437 23262 ts_tablet_manager.cc:1428] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 00:53:19.184892 23260 consensus_queue.cc:260] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:19.186125 23260 ts_tablet_manager.cc:1428] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 00:53:19.195637 23298 tablet_replica.cc:330] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947: stopping tablet replica
I20260809 00:53:19.195861 23295 tablet_replica.cc:330] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea: stopping tablet replica
I20260809 00:53:19.196249 23298 raft_consensus.cc:2229] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:19.196431 23295 raft_consensus.cc:2229] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:53:19.197253 23295 raft_consensus.cc:2258] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:19.197419 23298 raft_consensus.cc:2258] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:19.199853 23295 ts_tablet_manager.cc:1905] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:19.199854 23298 ts_tablet_manager.cc:1905] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:19.208171 23298 ts_tablet_manager.cc:1918] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:19.208174 23295 ts_tablet_manager.cc:1918] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:19.208406 23298 log.cc:1196] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/b5b24931cfed4a678af3d27a1b23f841
I20260809 00:53:19.208434 23295 log.cc:1196] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/b5b24931cfed4a678af3d27a1b23f841
I20260809 00:53:19.209035 23298 ts_tablet_manager.cc:1939] T b5b24931cfed4a678af3d27a1b23f841 P 92771243eee44cef9d4d1f7626f58947: Deleting consensus metadata
I20260809 00:53:19.209041 23295 ts_tablet_manager.cc:1939] T b5b24931cfed4a678af3d27a1b23f841 P 686eb4031a3f46e6be453b4799ca13ea: Deleting consensus metadata
I20260809 00:53:19.211313 22401 catalog_manager.cc:4841] TS 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685): tablet b5b24931cfed4a678af3d27a1b23f841 (table test-table [id=c5f23872b2354abfa845c14cf3ad87c1]) successfully deleted
I20260809 00:53:19.211318 22400 catalog_manager.cc:4841] TS 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509): tablet b5b24931cfed4a678af3d27a1b23f841 (table test-table [id=c5f23872b2354abfa845c14cf3ad87c1]) successfully deleted
I20260809 00:53:19.257709 23270 raft_consensus.cc:482] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:53:19.258054 23270 raft_consensus.cc:504] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:19.259727 23270 leader_election.cc:288] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685), 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509)
I20260809 00:53:19.268631 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.260480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:42759 (local address 127.9.225.1:40685)
0809 00:53:19.260685 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:19.260691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:19.260823 (+ 132us) server_negotiation.cc:407] Connection header received
0809 00:53:19.261190 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:19.261197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:19.261296 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:19.261415 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:19.262203 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.263368 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.266114 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.267449 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.267584 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.267701 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:19.268133 (+ 432us) server_negotiation.cc:299] Negotiation successful
0809 00:53:19.268321 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 00:53:19.268759 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.260051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:40685 (local address 127.9.225.3:42759)
0809 00:53:19.260674 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:19.260700 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:19.260987 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:19.261668 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:19.261673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:19.261700 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:19.261984 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.262000 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.263536 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.263543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.265897 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.265914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.267656 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.267663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.267908 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.267948 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 00:53:19.268066 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 00:53:19.268310 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":50,"thread_start_us":615,"threads_started":1}
I20260809 00:53:19.268812 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.260212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:39493 (local address 127.9.225.2:41509)
0809 00:53:19.260436 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:19.260444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:19.261478 (+ 1034us) server_negotiation.cc:407] Connection header received
0809 00:53:19.261684 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:19.261689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:19.261784 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:19.261896 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:19.262658 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.263815 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.265803 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.267403 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.267586 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.267695 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:19.268321 (+ 626us) server_negotiation.cc:299] Negotiation successful
0809 00:53:19.268476 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 00:53:19.269641 22661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74e0b4752e54fe9b08ae37bfef1a5bb" candidate_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92771243eee44cef9d4d1f7626f58947" is_pre_election: true
I20260809 00:53:19.269929 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.260060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:41509 (local address 127.9.225.3:39493)
0809 00:53:19.261324 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:19.261378 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:19.261533 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:19.261937 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:19.261942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:19.261973 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:19.262441 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.262455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.264009 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.264013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.265614 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.265629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.267895 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.267900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.268090 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.268156 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 00:53:19.269460 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0809 00:53:19.269609 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":419,"thread_start_us":494,"threads_started":1}
I20260809 00:53:19.269955 22661 raft_consensus.cc:2454] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d57d79c649a4a1fb38bb4a9c22ba752 in term 0.
I20260809 00:53:19.270576 23011 leader_election.cc:302] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d57d79c649a4a1fb38bb4a9c22ba752, 92771243eee44cef9d4d1f7626f58947; no voters:
I20260809 00:53:19.271134 23270 raft_consensus.cc:2781] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:53:19.271270 23270 raft_consensus.cc:482] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:53:19.271016 22900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74e0b4752e54fe9b08ae37bfef1a5bb" candidate_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686eb4031a3f46e6be453b4799ca13ea" is_pre_election: true
I20260809 00:53:19.271360 23270 raft_consensus.cc:3037] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:19.271586 22900 raft_consensus.cc:2454] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d57d79c649a4a1fb38bb4a9c22ba752 in term 0.
I20260809 00:53:19.275068 23270 raft_consensus.cc:504] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:19.276018 23270 leader_election.cc:288] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [CANDIDATE]: Term 1 election: Requested vote from peers 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685), 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509)
I20260809 00:53:19.276690 22900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74e0b4752e54fe9b08ae37bfef1a5bb" candidate_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686eb4031a3f46e6be453b4799ca13ea"
I20260809 00:53:19.276713 22661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c74e0b4752e54fe9b08ae37bfef1a5bb" candidate_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92771243eee44cef9d4d1f7626f58947"
I20260809 00:53:19.276947 22661 raft_consensus.cc:3037] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:19.276950 22900 raft_consensus.cc:3037] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:19.279874 22900 raft_consensus.cc:2454] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d57d79c649a4a1fb38bb4a9c22ba752 in term 1.
I20260809 00:53:19.280563 23009 leader_election.cc:302] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 686eb4031a3f46e6be453b4799ca13ea, 6d57d79c649a4a1fb38bb4a9c22ba752; no voters:
I20260809 00:53:19.280745 22661 raft_consensus.cc:2454] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d57d79c649a4a1fb38bb4a9c22ba752 in term 1.
I20260809 00:53:19.280956 23270 raft_consensus.cc:2781] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:19.281646 23270 raft_consensus.cc:686] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 LEADER]: Becoming Leader. State: Replica: 6d57d79c649a4a1fb38bb4a9c22ba752, State: Running, Role: LEADER
I20260809 00:53:19.282141 23270 consensus_queue.cc:237] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:19.286494 22422 catalog_manager.cc:5495] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d57d79c649a4a1fb38bb4a9c22ba752 (127.9.225.3). New cstate: current_term: 1 leader_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } health_report { overall_health: HEALTHY } } }
Aug 09 00:53:19 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236796, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.62@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:19.337846 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.320400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:41961 (local address 127.0.0.1:34332)
0809 00:53:19.321334 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:19.321400 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:19.321604 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:19.321955 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:19.321960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:19.322840 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:19.323172 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.323183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.324384 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.324391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.325463 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.325476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.326255 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.326260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.326466 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.327365 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:19.327394 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:19.330547 (+ 3153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:19.334731 (+ 4184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:19.334749 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:19.334758 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:19.335262 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:19.335776 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:19.335779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:19.335782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:19.335972 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:19.336800 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:19.336810 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:19.337004 (+ 194us) client_negotiation.cc:769] Sending connection context
0809 00:53:19.337166 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 00:53:19.337510 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":639,"threads_started":1}
I20260809 00:53:19.338442 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.320337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.9.225.62:41961)
0809 00:53:19.321177 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:19.321185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:19.321504 (+ 319us) server_negotiation.cc:407] Connection header received
0809 00:53:19.321736 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:19.321745 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:19.321805 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:19.321924 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:19.323373 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.324226 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.325618 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.326047 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.326201 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.330798 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:19.330803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:19.330807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:19.330843 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:19.334548 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:19.335424 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:19.335430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:19.335433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:19.335642 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:19.336107 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:19.336113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:19.336117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:19.336560 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:19.336726 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:19.337361 (+ 635us) server_negotiation.cc:299] Negotiation successful
0809 00:53:19.337728 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":639,"thread_start_us":445,"threads_started":1}
I20260809 00:53:19.344592 22422 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:34332:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 00:53:19.346374 22422 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:53:19.355028 23097 tablet_service.cc:1467] Processing CreateTablet for tablet 47985d49de8347b2b126a4c52690ad9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0c5415a500154fa0a244082c45f47e95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 00:53:19.355221 22859 tablet_service.cc:1467] Processing CreateTablet for tablet 47985d49de8347b2b126a4c52690ad9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0c5415a500154fa0a244082c45f47e95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 00:53:19.355728 23097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47985d49de8347b2b126a4c52690ad9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:19.355827 22859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47985d49de8347b2b126a4c52690ad9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:19.362053 22629 tablet_service.cc:1467] Processing CreateTablet for tablet 47985d49de8347b2b126a4c52690ad9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0c5415a500154fa0a244082c45f47e95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 00:53:19.362826 22629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47985d49de8347b2b126a4c52690ad9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:19.363855 23261 tablet_bootstrap.cc:492] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea: Bootstrap starting.
I20260809 00:53:19.365842 23261 tablet_bootstrap.cc:654] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:19.369161 23260 tablet_bootstrap.cc:492] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752: Bootstrap starting.
I20260809 00:53:19.370842 23260 tablet_bootstrap.cc:654] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:19.371436 23262 tablet_bootstrap.cc:492] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: Bootstrap starting.
I20260809 00:53:19.373445 23262 tablet_bootstrap.cc:654] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:19.374018 23261 tablet_bootstrap.cc:492] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea: No bootstrap required, opened a new log
I20260809 00:53:19.374217 23261 ts_tablet_manager.cc:1397] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20260809 00:53:19.375097 23261 raft_consensus.cc:348] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } }
I20260809 00:53:19.375407 23261 raft_consensus.cc:374] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:19.375509 23261 raft_consensus.cc:729] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 686eb4031a3f46e6be453b4799ca13ea, State: Initialized, Role: FOLLOWER
I20260809 00:53:19.375877 23262 tablet_bootstrap.cc:492] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: No bootstrap required, opened a new log
I20260809 00:53:19.375819 23261 consensus_queue.cc:260] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } }
I20260809 00:53:19.376055 23262 ts_tablet_manager.cc:1397] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260809 00:53:19.376864 23262 raft_consensus.cc:348] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } }
I20260809 00:53:19.377127 23262 raft_consensus.cc:374] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:19.376845 23287 tablet_replica.cc:439] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } }
I20260809 00:53:19.377211 23262 raft_consensus.cc:729] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92771243eee44cef9d4d1f7626f58947, State: Initialized, Role: FOLLOWER
I20260809 00:53:19.377264 23287 tablet_replica.cc:442] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:53:19.377522 23262 consensus_queue.cc:260] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } }
I20260809 00:53:19.378039 23272 tablet_replica.cc:439] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } }
I20260809 00:53:19.378636 23272 tablet_replica.cc:442] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:53:19.379199 23261 ts_tablet_manager.cc:1428] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260809 00:53:19.379974 23262 ts_tablet_manager.cc:1428] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260809 00:53:19.381418 23260 tablet_bootstrap.cc:492] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752: No bootstrap required, opened a new log
I20260809 00:53:19.381719 23260 ts_tablet_manager.cc:1397] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260809 00:53:19.382936 23260 raft_consensus.cc:348] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } }
I20260809 00:53:19.383351 23260 raft_consensus.cc:374] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:19.383518 23260 raft_consensus.cc:729] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d57d79c649a4a1fb38bb4a9c22ba752, State: Initialized, Role: FOLLOWER
I20260809 00:53:19.384379 23260 consensus_queue.cc:260] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } }
I20260809 00:53:19.385130 23260 ts_tablet_manager.cc:1428] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 00:53:19.384970 23270 tablet_replica.cc:439] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } }
I20260809 00:53:19.386101 23270 tablet_replica.cc:442] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:53:19.696332 23270 consensus_queue.cc:1035] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:19.703689 23305 consensus_queue.cc:1035] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:19.731796 23272 raft_consensus.cc:482] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:53:19.732121 23272 raft_consensus.cc:504] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } }
I20260809 00:53:19.734146 23272 leader_election.cc:288] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d57d79c649a4a1fb38bb4a9c22ba752 (127.9.225.3:36925), 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509)
I20260809 00:53:19.744210 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.734625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:43983 (local address 127.9.225.2:41509)
0809 00:53:19.734881 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:19.734891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:19.735875 (+ 984us) server_negotiation.cc:407] Connection header received
0809 00:53:19.735991 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:19.735996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:19.736080 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:19.736178 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:19.737106 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.738326 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.741506 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.742796 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.742979 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.743108 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:19.743739 (+ 631us) server_negotiation.cc:299] Negotiation successful
0809 00:53:19.743897 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 00:53:19.744270 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.734572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:60155 (local address 127.9.225.3:36925)
0809 00:53:19.735333 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:19.735340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:19.735725 (+ 385us) server_negotiation.cc:407] Connection header received
0809 00:53:19.735956 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:19.735961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:19.736047 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:19.736178 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:19.737106 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.738346 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.741499 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.742868 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.743035 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.743139 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:19.743732 (+ 593us) server_negotiation.cc:299] Negotiation successful
0809 00:53:19.743897 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":471,"threads_started":1}
I20260809 00:53:19.744498 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.734572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:36925 (local address 127.9.225.1:60155)
0809 00:53:19.735531 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:19.735557 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:19.735773 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:19.736217 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:19.736222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:19.736261 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:19.736910 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.736925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.738524 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.738530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.741224 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.741246 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.743163 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.743173 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.743487 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.743570 (+ 83us) client_negotiation.cc:769] Sending connection context
0809 00:53:19.743680 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 00:53:19.743924 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":293,"spinlock_wait_cycles":896,"thread_start_us":655,"threads_started":1}
I20260809 00:53:19.744521 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.734583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:41509 (local address 127.9.225.1:43983)
0809 00:53:19.735752 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:19.735770 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:19.735886 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:19.736217 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:19.736222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:19.736266 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:19.736864 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.736885 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.738524 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.738530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.741221 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.741246 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.743202 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.743207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.743463 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.743565 (+ 102us) client_negotiation.cc:769] Sending connection context
0809 00:53:19.743680 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 00:53:19.743924 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":383,"spinlock_wait_cycles":4352,"thread_start_us":817,"threads_started":1}
I20260809 00:53:19.745334 23141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47985d49de8347b2b126a4c52690ad9d" candidate_uuid: "92771243eee44cef9d4d1f7626f58947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" is_pre_election: true
I20260809 00:53:19.745411 22900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47985d49de8347b2b126a4c52690ad9d" candidate_uuid: "92771243eee44cef9d4d1f7626f58947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686eb4031a3f46e6be453b4799ca13ea" is_pre_election: true
I20260809 00:53:19.745675 22900 raft_consensus.cc:2454] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92771243eee44cef9d4d1f7626f58947 in term 0.
I20260809 00:53:19.745667 23141 raft_consensus.cc:2454] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92771243eee44cef9d4d1f7626f58947 in term 0.
I20260809 00:53:19.746235 22559 leader_election.cc:302] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 686eb4031a3f46e6be453b4799ca13ea, 6d57d79c649a4a1fb38bb4a9c22ba752, 92771243eee44cef9d4d1f7626f58947; no voters:
I20260809 00:53:19.746771 23272 raft_consensus.cc:2781] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:53:19.746896 23272 raft_consensus.cc:482] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:53:19.746974 23272 raft_consensus.cc:3037] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:19.749174 23272 raft_consensus.cc:504] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } }
I20260809 00:53:19.749888 23272 leader_election.cc:288] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [CANDIDATE]: Term 1 election: Requested vote from peers 6d57d79c649a4a1fb38bb4a9c22ba752 (127.9.225.3:36925), 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509)
I20260809 00:53:19.750463 22900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47985d49de8347b2b126a4c52690ad9d" candidate_uuid: "92771243eee44cef9d4d1f7626f58947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686eb4031a3f46e6be453b4799ca13ea"
I20260809 00:53:19.750489 23141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47985d49de8347b2b126a4c52690ad9d" candidate_uuid: "92771243eee44cef9d4d1f7626f58947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752"
I20260809 00:53:19.750715 22900 raft_consensus.cc:3037] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:19.750751 23141 raft_consensus.cc:3037] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:19.752948 23141 raft_consensus.cc:2454] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92771243eee44cef9d4d1f7626f58947 in term 1.
I20260809 00:53:19.753001 22900 raft_consensus.cc:2454] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92771243eee44cef9d4d1f7626f58947 in term 1.
I20260809 00:53:19.753546 22559 leader_election.cc:302] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d57d79c649a4a1fb38bb4a9c22ba752, 92771243eee44cef9d4d1f7626f58947; no voters:
I20260809 00:53:19.753926 23272 raft_consensus.cc:2781] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:19.754439 23272 raft_consensus.cc:686] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [term 1 LEADER]: Becoming Leader. State: Replica: 92771243eee44cef9d4d1f7626f58947, State: Running, Role: LEADER
I20260809 00:53:19.754935 23272 consensus_queue.cc:237] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } }
I20260809 00:53:19.757159 23342 tablet_replica.cc:439] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: TxnStatusTablet state changed. Reason: New leader 92771243eee44cef9d4d1f7626f58947. Latest consensus state: current_term: 1 leader_uuid: "92771243eee44cef9d4d1f7626f58947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } }
I20260809 00:53:19.757618 23342 tablet_replica.cc:442] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: This TxnStatusTablet replica's current role is: LEADER
I20260809 00:53:19.758687 23345 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 00:53:19.758955 23345 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 00:53:19.759688 22419 catalog_manager.cc:5495] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92771243eee44cef9d4d1f7626f58947 (127.9.225.1). New cstate: current_term: 1 leader_uuid: "92771243eee44cef9d4d1f7626f58947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } health_report { overall_health: UNKNOWN } } }
I20260809 00:53:19.843796 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.836526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.9.225.1:40685)
0809 00:53:19.837292 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:19.837297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:19.837316 (+ 19us) server_negotiation.cc:407] Connection header received
0809 00:53:19.837471 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:19.837475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:19.837568 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:19.837710 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:53:19.838213 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.839113 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.841130 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.841681 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.841788 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.842382 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:53:19.843073 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:53:19.843188 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:19.843435 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 00:53:19.843525 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":450,"threads_started":1}
I20260809 00:53:19.843878 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.836505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:40685 (local address 127.0.0.1:43500)
0809 00:53:19.836824 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:19.836847 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:19.837019 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:19.837788 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:19.837793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:19.837809 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 00:53:19.838034 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.838048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.839272 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.839281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.840950 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.840969 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.841874 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.841881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.842074 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.842172 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 00:53:19.843196 (+ 1024us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 00:53:19.843226 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 00:53:19.843342 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 00:53:19.843529 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 00:53:19.848990 22900 raft_consensus.cc:1261] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Refusing update from remote peer 92771243eee44cef9d4d1f7626f58947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:19.849040 23141 raft_consensus.cc:1261] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Refusing update from remote peer 92771243eee44cef9d4d1f7626f58947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:19.849984 23342 consensus_queue.cc:1035] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:19.850392 23272 consensus_queue.cc:1035] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:19.854202 23330 tablet_replica.cc:439] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752: TxnStatusTablet state changed. Reason: New leader 92771243eee44cef9d4d1f7626f58947. Latest consensus state: current_term: 1 leader_uuid: "92771243eee44cef9d4d1f7626f58947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } }
I20260809 00:53:19.854602 23330 tablet_replica.cc:442] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:53:19.856894 23287 tablet_replica.cc:439] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea: TxnStatusTablet state changed. Reason: New leader 92771243eee44cef9d4d1f7626f58947. Latest consensus state: current_term: 1 leader_uuid: "92771243eee44cef9d4d1f7626f58947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } }
I20260809 00:53:19.857249 23287 tablet_replica.cc:442] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:53:19.860702 23328 mvcc.cc:205] Tried to move back new op lower bound from 7316425932173152256 to 7316425931800813568. Current Snapshot: MvccSnapshot[applied={T|T < 7316425932173152256}]
I20260809 00:53:19.868629 23287 tablet_replica.cc:439] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92771243eee44cef9d4d1f7626f58947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } }
I20260809 00:53:19.868991 23287 tablet_replica.cc:442] T 47985d49de8347b2b126a4c52690ad9d P 686eb4031a3f46e6be453b4799ca13ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:53:19.869866 23272 tablet_replica.cc:439] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92771243eee44cef9d4d1f7626f58947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } }
I20260809 00:53:19.870278 23272 tablet_replica.cc:442] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: This TxnStatusTablet replica's current role is: LEADER
I20260809 00:53:19.889915 23330 tablet_replica.cc:439] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92771243eee44cef9d4d1f7626f58947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } }
I20260809 00:53:19.890250 23330 tablet_replica.cc:442] T 47985d49de8347b2b126a4c52690ad9d P 6d57d79c649a4a1fb38bb4a9c22ba752: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 00:53:19.892269 23361 tablet_replica.cc:439] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92771243eee44cef9d4d1f7626f58947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } } peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } }
I20260809 00:53:19.892741 23361 tablet_replica.cc:442] T 47985d49de8347b2b126a4c52690ad9d P 92771243eee44cef9d4d1f7626f58947: This TxnStatusTablet replica's current role is: LEADER
I20260809 00:53:19.900979 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.883362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.9.225.3:36925)
0809 00:53:19.883648 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:19.883658 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:19.894260 (+ 10602us) server_negotiation.cc:407] Connection header received
0809 00:53:19.895022 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:19.895029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:19.895153 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:19.895328 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:53:19.896116 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.896884 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.898548 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.899043 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.899199 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.899661 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:53:19.900310 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:53:19.900513 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:19.900569 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 00:53:19.900672 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 00:53:19.919106 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.913137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:39879 (local address 127.9.225.1:40685)
0809 00:53:19.913331 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:19.913338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:19.914025 (+ 687us) server_negotiation.cc:407] Connection header received
0809 00:53:19.914270 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:19.914274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:19.914347 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:19.914462 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:53:19.914891 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.915679 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.917131 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.917669 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.917801 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.918315 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:53:19.918423 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:53:19.918531 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:19.918677 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 00:53:19.918765 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 00:53:19.919186 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.913102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:40685 (local address 127.9.225.3:39879)
0809 00:53:19.913905 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:19.913922 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:19.914117 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:19.914476 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:19.914481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:19.914493 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 00:53:19.914733 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.914742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.915842 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.915854 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.916975 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.916988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.917887 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.917893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.918033 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.918101 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 00:53:19.918535 (+ 434us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 00:53:19.918550 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 00:53:19.918654 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 00:53:19.918806 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":463,"threads_started":1}
I20260809 00:53:19.967164 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.957912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:57015 (local address 127.9.225.3:36925)
0809 00:53:19.958094 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:19.958102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:19.959243 (+ 1141us) server_negotiation.cc:407] Connection header received
0809 00:53:19.959570 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:19.959576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:19.959684 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:19.959751 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:53:19.961527 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.962767 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.964781 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.965503 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.965648 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.966334 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:53:19.966524 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:53:19.966627 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:19.966798 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 00:53:19.966890 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 00:53:19.967429 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:19.957444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:36925 (local address 127.9.225.1:57015)
0809 00:53:19.959109 (+ 1665us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:19.959141 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:19.959379 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:19.960191 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:19.960197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:19.960212 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 00:53:19.961227 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.961245 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.962944 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.962948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.964598 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:19.964617 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:19.965722 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:19.965729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:19.965959 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:19.966056 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 00:53:19.966650 (+ 594us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 00:53:19.966665 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 00:53:19.966763 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 00:53:19.966949 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1429,"thread_start_us":870,"threads_started":1}
I20260809 00:53:20.014599 22419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:20.014928 22419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:20.019227 22419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448: Sending DeleteTablet for 3 replicas of tablet c74e0b4752e54fe9b08ae37bfef1a5bb
I20260809 00:53:20.020664 22859 tablet_service.cc:1514] Processing DeleteTablet for tablet c74e0b4752e54fe9b08ae37bfef1a5bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:20 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:32960
I20260809 00:53:20.020659 23097 tablet_service.cc:1514] Processing DeleteTablet for tablet c74e0b4752e54fe9b08ae37bfef1a5bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:20 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:40828
I20260809 00:53:20.021457 22629 tablet_service.cc:1514] Processing DeleteTablet for tablet c74e0b4752e54fe9b08ae37bfef1a5bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:20 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:43494
I20260809 00:53:20.031688 23380 tablet_replica.cc:330] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea: stopping tablet replica
I20260809 00:53:20.032338 23381 tablet_replica.cc:330] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752: stopping tablet replica
I20260809 00:53:20.032721 23381 raft_consensus.cc:2229] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:53:20.033133 23380 raft_consensus.cc:2229] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:20.033951 23381 raft_consensus.cc:2258] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:20.034340 23380 raft_consensus.cc:2258] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:20.035912 23381 ts_tablet_manager.cc:1905] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:20.042434 23380 ts_tablet_manager.cc:1905] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:20.044776 23381 ts_tablet_manager.cc:1918] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:53:20.045094 23381 log.cc:1196] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/c74e0b4752e54fe9b08ae37bfef1a5bb
I20260809 00:53:20.045713 23381 ts_tablet_manager.cc:1939] T c74e0b4752e54fe9b08ae37bfef1a5bb P 6d57d79c649a4a1fb38bb4a9c22ba752: Deleting consensus metadata
I20260809 00:53:20.047233 22401 catalog_manager.cc:4841] TS 6d57d79c649a4a1fb38bb4a9c22ba752 (127.9.225.3:36925): tablet c74e0b4752e54fe9b08ae37bfef1a5bb (table test-table [id=d9e389a01d2042b7b01c0725625e3007]) successfully deleted
I20260809 00:53:20.058385 23380 ts_tablet_manager.cc:1918] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:53:20.059175 23383 tablet_replica.cc:330] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947: stopping tablet replica
I20260809 00:53:20.059609 23383 raft_consensus.cc:2229] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:20.060051 23383 raft_consensus.cc:2258] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:20.061517 23383 ts_tablet_manager.cc:1905] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:20.068056 23380 log.cc:1196] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/c74e0b4752e54fe9b08ae37bfef1a5bb
I20260809 00:53:20.068907 23380 ts_tablet_manager.cc:1939] T c74e0b4752e54fe9b08ae37bfef1a5bb P 686eb4031a3f46e6be453b4799ca13ea: Deleting consensus metadata
I20260809 00:53:20.070987 22400 catalog_manager.cc:4841] TS 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509): tablet c74e0b4752e54fe9b08ae37bfef1a5bb (table test-table [id=d9e389a01d2042b7b01c0725625e3007]) successfully deleted
I20260809 00:53:20.073137 23383 ts_tablet_manager.cc:1918] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:53:20.073431 23383 log.cc:1196] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/c74e0b4752e54fe9b08ae37bfef1a5bb
I20260809 00:53:20.074090 23383 ts_tablet_manager.cc:1939] T c74e0b4752e54fe9b08ae37bfef1a5bb P 92771243eee44cef9d4d1f7626f58947: Deleting consensus metadata
I20260809 00:53:20.075744 22401 catalog_manager.cc:4841] TS 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685): tablet c74e0b4752e54fe9b08ae37bfef1a5bb (table test-table [id=d9e389a01d2042b7b01c0725625e3007]) successfully deleted
Aug 09 00:53:20 dist-test-slave-j2vl krb5kdc[22341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 00:53:20 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:20 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 00:53:20.150518 23395 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:20.117863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.9.225.62:41961)
0809 00:53:20.130421 (+ 12558us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:20.130431 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:20.131384 (+ 953us) server_negotiation.cc:407] Connection header received
0809 00:53:20.131849 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:20.131858 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:20.131960 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:20.132058 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:20.134585 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:20.135963 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:20.138047 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:20.138777 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:20.138946 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:20.150137 (+ 11191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34344: BlockingRecv error: recv got EOF from 127.0.0.1:34344 (error 108)
Metrics: {"server-negotiator.queue_time_us":12343,"thread_start_us":12643,"threads_started":1}
Aug 09 00:53:20 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM
I20260809 00:53:20.173491 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:20.150614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.9.225.62:41961)
0809 00:53:20.151126 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:20.151132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:20.151704 (+ 572us) server_negotiation.cc:407] Connection header received
0809 00:53:20.152020 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:20.152026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:20.152120 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:20.152203 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:20.154462 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:20.155713 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:20.157784 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:20.158467 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:20.158590 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:20.165504 (+ 6914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:20.165514 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:20.165520 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:20.165575 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:20.169679 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:20.170839 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:20.170846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:20.170850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:20.171048 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:20.171569 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:20.171574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:20.171578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:20.172035 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:20.172417 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:20.172711 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 00:53:20.172978 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":13249,"threads_started":1}
I20260809 00:53:20.178476 22419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:53:20.179337 22419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:53:20.190083 22859 tablet_service.cc:1467] Processing CreateTablet for tablet c2f27a3c10e74fad9b638edf77f7e566 (DEFAULT_TABLE table=test-table [id=e41ad3adbaaf4da5988e690182a35b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:20.190428 22629 tablet_service.cc:1467] Processing CreateTablet for tablet c2f27a3c10e74fad9b638edf77f7e566 (DEFAULT_TABLE table=test-table [id=e41ad3adbaaf4da5988e690182a35b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:20.190670 22859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f27a3c10e74fad9b638edf77f7e566. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:20.190940 22629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f27a3c10e74fad9b638edf77f7e566. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:20.198441 23097 tablet_service.cc:1467] Processing CreateTablet for tablet c2f27a3c10e74fad9b638edf77f7e566 (DEFAULT_TABLE table=test-table [id=e41ad3adbaaf4da5988e690182a35b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:20.198854 23403 tablet_bootstrap.cc:492] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea: Bootstrap starting.
I20260809 00:53:20.199020 23097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2f27a3c10e74fad9b638edf77f7e566. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:20.200615 23403 tablet_bootstrap.cc:654] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:20.202925 23403 tablet_bootstrap.cc:492] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea: No bootstrap required, opened a new log
I20260809 00:53:20.203106 23403 ts_tablet_manager.cc:1397] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 00:53:20.203982 23403 raft_consensus.cc:348] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.204265 23403 raft_consensus.cc:374] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:20.204367 23403 raft_consensus.cc:729] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 686eb4031a3f46e6be453b4799ca13ea, State: Initialized, Role: FOLLOWER
I20260809 00:53:20.204648 23403 consensus_queue.cc:260] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.205479 23403 ts_tablet_manager.cc:1428] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 00:53:20.218005 23405 tablet_bootstrap.cc:492] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947: Bootstrap starting.
I20260809 00:53:20.218005 23406 tablet_bootstrap.cc:492] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752: Bootstrap starting.
I20260809 00:53:20.220687 23405 tablet_bootstrap.cc:654] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:20.221886 23406 tablet_bootstrap.cc:654] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:20.231837 23406 tablet_bootstrap.cc:492] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752: No bootstrap required, opened a new log
I20260809 00:53:20.232125 23406 ts_tablet_manager.cc:1397] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20260809 00:53:20.233175 23406 raft_consensus.cc:348] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.233522 23406 raft_consensus.cc:374] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:20.233722 23406 raft_consensus.cc:729] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d57d79c649a4a1fb38bb4a9c22ba752, State: Initialized, Role: FOLLOWER
I20260809 00:53:20.234082 23406 consensus_queue.cc:260] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.239893 23406 ts_tablet_manager.cc:1428] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260809 00:53:20.244993 23405 tablet_bootstrap.cc:492] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947: No bootstrap required, opened a new log
I20260809 00:53:20.245249 23405 ts_tablet_manager.cc:1397] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.021s
I20260809 00:53:20.246250 23405 raft_consensus.cc:348] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.246553 23405 raft_consensus.cc:374] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:20.246680 23405 raft_consensus.cc:729] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92771243eee44cef9d4d1f7626f58947, State: Initialized, Role: FOLLOWER
I20260809 00:53:20.247035 23405 consensus_queue.cc:260] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.249862 23405 ts_tablet_manager.cc:1428] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 00:53:20.343585 23305 raft_consensus.cc:482] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:53:20.343886 23305 raft_consensus.cc:504] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.344748 23305 leader_election.cc:288] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685), 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509)
I20260809 00:53:20.345443 22661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f27a3c10e74fad9b638edf77f7e566" candidate_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92771243eee44cef9d4d1f7626f58947" is_pre_election: true
I20260809 00:53:20.345620 22899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f27a3c10e74fad9b638edf77f7e566" candidate_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686eb4031a3f46e6be453b4799ca13ea" is_pre_election: true
I20260809 00:53:20.345765 22661 raft_consensus.cc:2454] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d57d79c649a4a1fb38bb4a9c22ba752 in term 0.
I20260809 00:53:20.345908 22899 raft_consensus.cc:2454] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d57d79c649a4a1fb38bb4a9c22ba752 in term 0.
I20260809 00:53:20.346469 23011 leader_election.cc:302] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 686eb4031a3f46e6be453b4799ca13ea, 6d57d79c649a4a1fb38bb4a9c22ba752, 92771243eee44cef9d4d1f7626f58947; no voters:
I20260809 00:53:20.346843 23305 raft_consensus.cc:2781] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:53:20.346979 23305 raft_consensus.cc:482] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:53:20.347114 23305 raft_consensus.cc:3037] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:20.350842 23305 raft_consensus.cc:504] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.351704 23305 leader_election.cc:288] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [CANDIDATE]: Term 1 election: Requested vote from peers 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685), 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509)
I20260809 00:53:20.352335 22661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f27a3c10e74fad9b638edf77f7e566" candidate_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92771243eee44cef9d4d1f7626f58947"
I20260809 00:53:20.352388 22899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2f27a3c10e74fad9b638edf77f7e566" candidate_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686eb4031a3f46e6be453b4799ca13ea"
I20260809 00:53:20.352636 22661 raft_consensus.cc:3037] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:20.352664 22899 raft_consensus.cc:3037] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:20.355399 22661 raft_consensus.cc:2454] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d57d79c649a4a1fb38bb4a9c22ba752 in term 1.
I20260809 00:53:20.355871 22899 raft_consensus.cc:2454] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d57d79c649a4a1fb38bb4a9c22ba752 in term 1.
I20260809 00:53:20.356215 23011 leader_election.cc:302] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d57d79c649a4a1fb38bb4a9c22ba752, 92771243eee44cef9d4d1f7626f58947; no voters:
I20260809 00:53:20.356623 23305 raft_consensus.cc:2781] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:20.356873 23305 raft_consensus.cc:686] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 LEADER]: Becoming Leader. State: Replica: 6d57d79c649a4a1fb38bb4a9c22ba752, State: Running, Role: LEADER
I20260809 00:53:20.357184 23305 consensus_queue.cc:237] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.360800 22419 catalog_manager.cc:5495] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d57d79c649a4a1fb38bb4a9c22ba752 (127.9.225.3). New cstate: current_term: 1 leader_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } health_report { overall_health: HEALTHY } } }
I20260809 00:53:20.377298 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:20.371168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.9.225.3:36925)
0809 00:53:20.371370 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:20.371376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:20.371395 (+ 19us) server_negotiation.cc:407] Connection header received
0809 00:53:20.371472 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:20.371476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:20.371552 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:20.371666 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:53:20.372143 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:20.373053 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:20.375218 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:20.375748 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:20.375864 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:20.376490 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:53:20.376617 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:53:20.376689 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:20.376885 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 00:53:20.376969 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 00:53:20.380647 22661 raft_consensus.cc:1261] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Refusing update from remote peer 6d57d79c649a4a1fb38bb4a9c22ba752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:20.380650 22899 raft_consensus.cc:1261] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Refusing update from remote peer 6d57d79c649a4a1fb38bb4a9c22ba752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:20.381444 23330 consensus_queue.cc:1035] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:20.382231 23305 consensus_queue.cc:1035] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:20.398793 22417 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:20.399062 22417 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:20.404745 22417 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448: Sending DeleteTablet for 3 replicas of tablet c2f27a3c10e74fad9b638edf77f7e566
I20260809 00:53:20.405913 22629 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f27a3c10e74fad9b638edf77f7e566 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:20 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:43494
I20260809 00:53:20.406258 23383 tablet_replica.cc:330] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947: stopping tablet replica
I20260809 00:53:20.406349 22628 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f27a3c10e74fad9b638edf77f7e566 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:20 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:43494
I20260809 00:53:20.406332 23097 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f27a3c10e74fad9b638edf77f7e566 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:20 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:40828
I20260809 00:53:20.406495 23383 raft_consensus.cc:2229] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:20.406652 23381 tablet_replica.cc:330] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752: stopping tablet replica
I20260809 00:53:20.406659 22859 tablet_service.cc:1514] Processing DeleteTablet for tablet c2f27a3c10e74fad9b638edf77f7e566 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:20 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:32960
I20260809 00:53:20.406767 23383 raft_consensus.cc:2258] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:20.406944 23380 tablet_replica.cc:330] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea: stopping tablet replica
I20260809 00:53:20.406862 23381 raft_consensus.cc:2229] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:53:20.407160 23380 raft_consensus.cc:2229] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:20.407392 23381 raft_consensus.cc:2258] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:20.407461 23380 raft_consensus.cc:2258] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:20.408718 23380 ts_tablet_manager.cc:1905] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:20.410725 22420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:53:20.411584 22420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:53:20.413168 23383 ts_tablet_manager.cc:1905] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:20.416786 23380 ts_tablet_manager.cc:1918] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:20.416992 23380 log.cc:1196] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/c2f27a3c10e74fad9b638edf77f7e566
I20260809 00:53:20.417546 23380 ts_tablet_manager.cc:1939] T c2f27a3c10e74fad9b638edf77f7e566 P 686eb4031a3f46e6be453b4799ca13ea: Deleting consensus metadata
I20260809 00:53:20.418955 22400 catalog_manager.cc:4841] TS 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509): tablet c2f27a3c10e74fad9b638edf77f7e566 (table test-table [id=e41ad3adbaaf4da5988e690182a35b2d]) successfully deleted
I20260809 00:53:20.420106 23381 ts_tablet_manager.cc:1905] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:20.421299 22628 tablet_service.cc:1467] Processing CreateTablet for tablet 43beb24d0fde4fdb84b419772dfbc463 (DEFAULT_TABLE table=test-table [id=c58df858aee845c4bdcb18586d440769]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:20.421840 22628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43beb24d0fde4fdb84b419772dfbc463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:20.426036 23383 ts_tablet_manager.cc:1918] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:20.426033 23097 tablet_service.cc:1467] Processing CreateTablet for tablet 43beb24d0fde4fdb84b419772dfbc463 (DEFAULT_TABLE table=test-table [id=c58df858aee845c4bdcb18586d440769]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:20.426268 23383 log.cc:1196] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/c2f27a3c10e74fad9b638edf77f7e566
I20260809 00:53:20.426571 23097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43beb24d0fde4fdb84b419772dfbc463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:20.427209 23383 ts_tablet_manager.cc:1939] T c2f27a3c10e74fad9b638edf77f7e566 P 92771243eee44cef9d4d1f7626f58947: Deleting consensus metadata
I20260809 00:53:20.428699 22401 catalog_manager.cc:4841] TS 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685): tablet c2f27a3c10e74fad9b638edf77f7e566 (table test-table [id=e41ad3adbaaf4da5988e690182a35b2d]) successfully deleted
W20260809 00:53:20.430027 22401 catalog_manager.cc:4808] TS 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685): delete failed for tablet c2f27a3c10e74fad9b638edf77f7e566 because the tablet was not found. No further retry: Not found: Tablet not found: c2f27a3c10e74fad9b638edf77f7e566
I20260809 00:53:20.430482 22859 tablet_service.cc:1467] Processing CreateTablet for tablet 43beb24d0fde4fdb84b419772dfbc463 (DEFAULT_TABLE table=test-table [id=c58df858aee845c4bdcb18586d440769]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:20.430995 22859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43beb24d0fde4fdb84b419772dfbc463. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:20.434180 23405 tablet_bootstrap.cc:492] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947: Bootstrap starting.
I20260809 00:53:20.436228 23405 tablet_bootstrap.cc:654] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:20.437418 23406 tablet_bootstrap.cc:492] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752: Bootstrap starting.
I20260809 00:53:20.437919 23381 ts_tablet_manager.cc:1918] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:20.438230 23381 log.cc:1196] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/c2f27a3c10e74fad9b638edf77f7e566
I20260809 00:53:20.438398 23403 tablet_bootstrap.cc:492] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea: Bootstrap starting.
I20260809 00:53:20.438941 23381 ts_tablet_manager.cc:1939] T c2f27a3c10e74fad9b638edf77f7e566 P 6d57d79c649a4a1fb38bb4a9c22ba752: Deleting consensus metadata
I20260809 00:53:20.439760 23403 tablet_bootstrap.cc:654] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:20.440222 22401 catalog_manager.cc:4841] TS 6d57d79c649a4a1fb38bb4a9c22ba752 (127.9.225.3:36925): tablet c2f27a3c10e74fad9b638edf77f7e566 (table test-table [id=e41ad3adbaaf4da5988e690182a35b2d]) successfully deleted
I20260809 00:53:20.441906 23406 tablet_bootstrap.cc:654] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:20.444147 23405 tablet_bootstrap.cc:492] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947: No bootstrap required, opened a new log
I20260809 00:53:20.444397 23405 ts_tablet_manager.cc:1397] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20260809 00:53:20.445544 23405 raft_consensus.cc:348] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.445922 23405 raft_consensus.cc:374] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:20.446020 23405 raft_consensus.cc:729] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92771243eee44cef9d4d1f7626f58947, State: Initialized, Role: FOLLOWER
I20260809 00:53:20.448295 23405 consensus_queue.cc:260] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.449563 23405 ts_tablet_manager.cc:1428] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 00:53:20.451550 23406 tablet_bootstrap.cc:492] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752: No bootstrap required, opened a new log
I20260809 00:53:20.451828 23406 ts_tablet_manager.cc:1397] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260809 00:53:20.452788 23406 raft_consensus.cc:348] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.453161 23406 raft_consensus.cc:374] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:20.453266 23406 raft_consensus.cc:729] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d57d79c649a4a1fb38bb4a9c22ba752, State: Initialized, Role: FOLLOWER
I20260809 00:53:20.453614 23406 consensus_queue.cc:260] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.454305 23406 ts_tablet_manager.cc:1428] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260809 00:53:20.465639 23403 tablet_bootstrap.cc:492] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea: No bootstrap required, opened a new log
I20260809 00:53:20.465998 23403 ts_tablet_manager.cc:1397] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.005s
I20260809 00:53:20.467280 23403 raft_consensus.cc:348] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.467648 23403 raft_consensus.cc:374] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:20.467790 23403 raft_consensus.cc:729] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 686eb4031a3f46e6be453b4799ca13ea, State: Initialized, Role: FOLLOWER
I20260809 00:53:20.468156 23403 consensus_queue.cc:260] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.479074 23403 ts_tablet_manager.cc:1428] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20260809 00:53:20.587258 23330 raft_consensus.cc:482] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:53:20.587725 23330 raft_consensus.cc:504] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.588706 23330 leader_election.cc:288] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685), 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509)
I20260809 00:53:20.589787 22899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43beb24d0fde4fdb84b419772dfbc463" candidate_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686eb4031a3f46e6be453b4799ca13ea" is_pre_election: true
I20260809 00:53:20.589854 22661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43beb24d0fde4fdb84b419772dfbc463" candidate_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92771243eee44cef9d4d1f7626f58947" is_pre_election: true
I20260809 00:53:20.590161 22899 raft_consensus.cc:2454] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d57d79c649a4a1fb38bb4a9c22ba752 in term 0.
I20260809 00:53:20.590161 22661 raft_consensus.cc:2454] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d57d79c649a4a1fb38bb4a9c22ba752 in term 0.
I20260809 00:53:20.590818 23011 leader_election.cc:302] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d57d79c649a4a1fb38bb4a9c22ba752, 92771243eee44cef9d4d1f7626f58947; no voters:
I20260809 00:53:20.591259 23330 raft_consensus.cc:2781] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:53:20.591392 23330 raft_consensus.cc:482] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:53:20.591511 23330 raft_consensus.cc:3037] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:20.595657 23330 raft_consensus.cc:504] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.596570 23330 leader_election.cc:288] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [CANDIDATE]: Term 1 election: Requested vote from peers 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685), 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509)
I20260809 00:53:20.597565 22899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43beb24d0fde4fdb84b419772dfbc463" candidate_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686eb4031a3f46e6be453b4799ca13ea"
I20260809 00:53:20.597549 22661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43beb24d0fde4fdb84b419772dfbc463" candidate_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92771243eee44cef9d4d1f7626f58947"
I20260809 00:53:20.597893 22899 raft_consensus.cc:3037] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:20.597946 22661 raft_consensus.cc:3037] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:20.600952 22899 raft_consensus.cc:2454] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d57d79c649a4a1fb38bb4a9c22ba752 in term 1.
I20260809 00:53:20.601094 22661 raft_consensus.cc:2454] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d57d79c649a4a1fb38bb4a9c22ba752 in term 1.
I20260809 00:53:20.601742 23009 leader_election.cc:302] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 686eb4031a3f46e6be453b4799ca13ea, 6d57d79c649a4a1fb38bb4a9c22ba752; no voters:
I20260809 00:53:20.602156 23330 raft_consensus.cc:2781] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:20.602391 23330 raft_consensus.cc:686] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 LEADER]: Becoming Leader. State: Replica: 6d57d79c649a4a1fb38bb4a9c22ba752, State: Running, Role: LEADER
I20260809 00:53:20.602825 23330 consensus_queue.cc:237] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } }
I20260809 00:53:20.607039 22420 catalog_manager.cc:5495] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d57d79c649a4a1fb38bb4a9c22ba752 (127.9.225.3). New cstate: current_term: 1 leader_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d57d79c649a4a1fb38bb4a9c22ba752" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 36925 } health_report { overall_health: HEALTHY } } }
I20260809 00:53:20.672798 22899 raft_consensus.cc:1261] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Refusing update from remote peer 6d57d79c649a4a1fb38bb4a9c22ba752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:20.673081 22661 raft_consensus.cc:1261] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Refusing update from remote peer 6d57d79c649a4a1fb38bb4a9c22ba752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:20.673617 23330 consensus_queue.cc:1035] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "686eb4031a3f46e6be453b4799ca13ea" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:20.674058 23305 consensus_queue.cc:1035] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92771243eee44cef9d4d1f7626f58947" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:20.761837 22420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:20.762264 22420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:20.765820 22420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cce8029d33641c296e7c92ae5be7448: Sending DeleteTablet for 3 replicas of tablet 43beb24d0fde4fdb84b419772dfbc463
I20260809 00:53:20.766969 22859 tablet_service.cc:1514] Processing DeleteTablet for tablet 43beb24d0fde4fdb84b419772dfbc463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:20 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:32960
I20260809 00:53:20.767014 22628 tablet_service.cc:1514] Processing DeleteTablet for tablet 43beb24d0fde4fdb84b419772dfbc463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:20 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:43494
I20260809 00:53:20.767069 23097 tablet_service.cc:1514] Processing DeleteTablet for tablet 43beb24d0fde4fdb84b419772dfbc463 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:20 UTC) from {username='oryx', principal='oryx/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:40828
I20260809 00:53:20.767350 23380 tablet_replica.cc:330] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea: stopping tablet replica
I20260809 00:53:20.767369 23381 tablet_replica.cc:330] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752: stopping tablet replica
I20260809 00:53:20.767371 23383 tablet_replica.cc:330] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947: stopping tablet replica
I20260809 00:53:20.767553 23380 raft_consensus.cc:2229] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:20.767558 23381 raft_consensus.cc:2229] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:53:20.767558 23383 raft_consensus.cc:2229] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:20.767783 23383 raft_consensus.cc:2258] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:20.767791 23380 raft_consensus.cc:2258] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:20.767967 23381 raft_consensus.cc:2258] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:20.768908 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 22528
I20260809 00:53:20.769164 23381 ts_tablet_manager.cc:1905] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:20.769459 23383 ts_tablet_manager.cc:1905] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:20.769536 23380 ts_tablet_manager.cc:1905] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:20.776906 23383 ts_tablet_manager.cc:1918] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:53:20.777122 23383 log.cc:1196] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/43beb24d0fde4fdb84b419772dfbc463
I20260809 00:53:20.777662 23383 ts_tablet_manager.cc:1939] T 43beb24d0fde4fdb84b419772dfbc463 P 92771243eee44cef9d4d1f7626f58947: Deleting consensus metadata
I20260809 00:53:20.778534 23380 ts_tablet_manager.cc:1918] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:53:20.778781 23380 log.cc:1196] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/43beb24d0fde4fdb84b419772dfbc463
I20260809 00:53:20.779367 23380 ts_tablet_manager.cc:1939] T 43beb24d0fde4fdb84b419772dfbc463 P 686eb4031a3f46e6be453b4799ca13ea: Deleting consensus metadata
I20260809 00:53:20.781028 22400 catalog_manager.cc:4841] TS 686eb4031a3f46e6be453b4799ca13ea (127.9.225.2:41509): tablet 43beb24d0fde4fdb84b419772dfbc463 (table test-table [id=c58df858aee845c4bdcb18586d440769]) successfully deleted
I20260809 00:53:20.781028 22401 catalog_manager.cc:4841] TS 92771243eee44cef9d4d1f7626f58947 (127.9.225.1:40685): tablet 43beb24d0fde4fdb84b419772dfbc463 (table test-table [id=c58df858aee845c4bdcb18586d440769]) successfully deleted
I20260809 00:53:20.782248 23381 ts_tablet_manager.cc:1918] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 00:53:20.782388 23381 log.cc:1196] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/43beb24d0fde4fdb84b419772dfbc463
I20260809 00:53:20.782924 23381 ts_tablet_manager.cc:1939] T 43beb24d0fde4fdb84b419772dfbc463 P 6d57d79c649a4a1fb38bb4a9c22ba752: Deleting consensus metadata
I20260809 00:53:20.784207 22401 catalog_manager.cc:4841] TS 6d57d79c649a4a1fb38bb4a9c22ba752 (127.9.225.3:36925): tablet 43beb24d0fde4fdb84b419772dfbc463 (table test-table [id=c58df858aee845c4bdcb18586d440769]) successfully deleted
Aug 09 00:53:20 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.225.1@KRBTEST.COM
I20260809 00:53:20.797577 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:20.771115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.9.225.1:40685)
0809 00:53:20.780955 (+ 9840us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:20.780962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:20.780984 (+ 22us) server_negotiation.cc:407] Connection header received
0809 00:53:20.781047 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:20.781055 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:20.781146 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:20.781269 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:20.782872 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:20.783960 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:20.785332 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:20.785814 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:20.785969 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:20.790744 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:20.790765 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:20.790794 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:20.790842 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:20.794234 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:20.795145 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:20.795154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:20.795156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:20.795307 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:20.795756 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:20.795760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:20.795763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:20.796169 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:20.796528 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:20.796872 (+ 344us) server_negotiation.cc:299] Negotiation successful
0809 00:53:20.797138 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9699,"thread_start_us":9581,"threads_started":1}
I20260809 00:53:20.798415 22741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:20.932335 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 22528
I20260809 00:53:20.974260 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 22755
Aug 09 00:53:20 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.225.2@KRBTEST.COM
I20260809 00:53:20.991032 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:20.975150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.9.225.2:41509)
0809 00:53:20.975792 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:20.975797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:20.975815 (+ 18us) server_negotiation.cc:407] Connection header received
0809 00:53:20.975872 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:20.975876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:20.975945 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:20.976042 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:20.977338 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:20.978214 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:20.979670 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:20.980118 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:20.980216 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:20.984705 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:20.984724 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:20.984743 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:20.984788 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:20.987686 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:20.988456 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:20.988473 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:20.988476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:20.988644 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:20.989181 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:20.989186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:20.989188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:20.989905 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:20.990205 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:20.990499 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 00:53:20.990699 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":365,"threads_started":1}
I20260809 00:53:20.991750 22974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:21.123660 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 22755
I20260809 00:53:21.161134 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 22985
Aug 09 00:53:21 dist-test-slave-j2vl krb5kdc[22341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.225.3@KRBTEST.COM
I20260809 00:53:21.179198 23445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:21.162057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.9.225.3:36925)
0809 00:53:21.162671 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:21.162676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:21.162693 (+ 17us) server_negotiation.cc:407] Connection header received
0809 00:53:21.162748 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:21.162752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:21.162823 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:21.162962 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:21.164367 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:21.165122 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:21.166587 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:21.167029 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:21.167184 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:21.171998 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:21.172019 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:21.172039 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:21.172093 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:21.175527 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:21.176582 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:21.176595 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:21.176599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:21.176825 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:21.177372 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:21.177378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:21.177381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:21.177815 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:21.178164 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:21.178526 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 00:53:21.178786 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":314,"threads_started":1}
I20260809 00:53:21.179979 23212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:21.315486 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 22985
I20260809 00:53:21.352983 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 22368
I20260809 00:53:21.354538 22481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:21.479338 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 22368
2026-08-09T00:53:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7052 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:53:21 dist-test-slave-j2vl krb5kdc[23454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:53:21 dist-test-slave-j2vl krb5kdc[23454](info): set up 2 sockets
Aug 09 00:53:21 dist-test-slave-j2vl krb5kdc[23454](info): commencing operation
krb5kdc: starting...
W20260809 00:53:22.877074 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.329s user 0.005s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:53:23 dist-test-slave-j2vl krb5kdc[23454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:53:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:53:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:53:23.201838 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:44619
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:44619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:23.589968 23470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:23.590332 23470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:23.590504 23470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:23.598533 23470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:53:23.598682 23470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:23.598783 23470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:23.598893 23470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:53:23.598963 23470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:53:23.610074 23470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:44619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:44619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:23.611661 23470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:23.613209 23470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:23.627929 23479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:23.627929 23478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:23.627948 23483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:23.629151 23470 server_base.cc:1034] running on GCE node
I20260809 00:53:23.630235 23470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:23.632190 23470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:23.633477 23470 hybrid_clock.cc:648] HybridClock initialized: now 1786236803633406 us; error 53 us; skew 500 ppm
I20260809 00:53:23.633941 23470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T00:53:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2219 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:53:23 dist-test-slave-j2vl krb5kdc[23494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:53:23 dist-test-slave-j2vl krb5kdc[23494](info): set up 2 sockets
Aug 09 00:53:23 dist-test-slave-j2vl krb5kdc[23494](info): commencing operation
krb5kdc: starting...
W20260809 00:53:25.157176 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.390s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:53:25 dist-test-slave-j2vl krb5kdc[23494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:53:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:53:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:53:25.477607 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43919
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:43919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:25.865270 23515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:25.865928 23515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:25.866117 23515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:25.874624 23515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:53:25.874809 23515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:25.874864 23515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:25.874925 23515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:53:25.874972 23515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:53:25.885924 23515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:43919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:25.887794 23515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:25.889367 23515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:25.903066 23536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:25.903113 23527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:25.903172 23526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:25.904230 23515 server_base.cc:1034] running on GCE node
I20260809 00:53:25.905190 23515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:25.907253 23515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:25.908548 23515 hybrid_clock.cc:648] HybridClock initialized: now 1786236805908490 us; error 52 us; skew 500 ppm
I20260809 00:53:25.909017 23515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:25 dist-test-slave-j2vl krb5kdc[23494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236805, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:25.912672 23515 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:53:25.914620 23515 webserver.cc:460] Webserver started at http://127.9.225.62:36573/ using document root <none> and password file <none>
I20260809 00:53:25.915506 23515 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:25.915673 23515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:25.916050 23515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:25.919615 23515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "5057a020f0ee4412860f6c5c4bbe8461"
format_stamp: "Formatted at 2026-08-09 00:53:25 on dist-test-slave-j2vl"
server_key: "48cd0f43292cfc84e884dc3e4d4012e2"
server_key_iv: "ca79a51e649da1aa46add12d1da79908"
server_key_version: "encryptionkey@0"
I20260809 00:53:25.920536 23515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "5057a020f0ee4412860f6c5c4bbe8461"
format_stamp: "Formatted at 2026-08-09 00:53:25 on dist-test-slave-j2vl"
server_key: "48cd0f43292cfc84e884dc3e4d4012e2"
server_key_iv: "ca79a51e649da1aa46add12d1da79908"
server_key_version: "encryptionkey@0"
I20260809 00:53:25.926934 23515 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260809 00:53:25.931229 23546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:25.932854 23515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 00:53:25.933100 23515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "5057a020f0ee4412860f6c5c4bbe8461"
format_stamp: "Formatted at 2026-08-09 00:53:25 on dist-test-slave-j2vl"
server_key: "48cd0f43292cfc84e884dc3e4d4012e2"
server_key_iv: "ca79a51e649da1aa46add12d1da79908"
server_key_version: "encryptionkey@0"
I20260809 00:53:25.933386 23515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:25.955897 23515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:25.961905 23515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:25.962329 23515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:25.986138 23515 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:43919
I20260809 00:53:25.986176 23623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:43919 every 8 connection(s)
I20260809 00:53:25.987890 23515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:53:25.991752 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 23515
I20260809 00:53:25.992034 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:53:25.992574 23624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:25.992835 10116 external_mini_cluster.cc:1428] Setting key 62e725690306d6aec2aef614676a38c8
Aug 09 00:53:26 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:53:26.010110 23624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461: Bootstrap starting.
I20260809 00:53:26.014204 23624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:26.015617 23624 log.cc:824] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:26.019510 23624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461: No bootstrap required, opened a new log
I20260809 00:53:26.019871 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:25.995131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.9.225.62:43919)
0809 00:53:25.996098 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:25.996115 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:25.996167 (+ 52us) server_negotiation.cc:407] Connection header received
0809 00:53:25.998170 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:25.998203 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:25.998714 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:25.999139 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:26.001449 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:26.002906 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:26.004824 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:26.005717 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:26.005892 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:26.011371 (+ 5479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:26.011392 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:26.011411 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:26.011460 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:26.014956 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:26.015981 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:26.015989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:26.015996 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:26.016187 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:26.016710 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:26.016717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:26.016723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:26.017578 (+ 855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:26.017820 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:26.018317 (+ 497us) server_negotiation.cc:299] Negotiation successful
0809 00:53:26.018680 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":457,"threads_started":1}
I20260809 00:53:26.028260 23624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5057a020f0ee4412860f6c5c4bbe8461" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43919 } }
I20260809 00:53:26.028882 23624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:26.029067 23624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5057a020f0ee4412860f6c5c4bbe8461, State: Initialized, Role: FOLLOWER
I20260809 00:53:26.029829 23624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5057a020f0ee4412860f6c5c4bbe8461" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43919 } }
I20260809 00:53:26.030089 23624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:53:26.030200 23624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:53:26.030332 23624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:26.032830 23624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5057a020f0ee4412860f6c5c4bbe8461" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43919 } }
I20260809 00:53:26.033669 23624 leader_election.cc:302] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5057a020f0ee4412860f6c5c4bbe8461; no voters:
I20260809 00:53:26.034368 23624 leader_election.cc:288] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:53:26.034536 23635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:26.035297 23635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 1 LEADER]: Becoming Leader. State: Replica: 5057a020f0ee4412860f6c5c4bbe8461, State: Running, Role: LEADER
I20260809 00:53:26.035878 23635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5057a020f0ee4412860f6c5c4bbe8461" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43919 } }
I20260809 00:53:26.036800 23624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:53:26.039069 23641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5057a020f0ee4412860f6c5c4bbe8461. Latest consensus state: current_term: 1 leader_uuid: "5057a020f0ee4412860f6c5c4bbe8461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5057a020f0ee4412860f6c5c4bbe8461" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43919 } } }
I20260809 00:53:26.039547 23641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:26.041474 23649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:53:26.040381 23637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5057a020f0ee4412860f6c5c4bbe8461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5057a020f0ee4412860f6c5c4bbe8461" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43919 } } }
I20260809 00:53:26.041770 23637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:26.047397 23649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:53:26.062579 23649 catalog_manager.cc:1347] Generated new cluster ID: a5cc9e49ec4746abb1eb130d0997d7fb
I20260809 00:53:26.062754 23649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:53:26.088461 23649 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:53:26.090143 23649 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:53:26.098235 23649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461: Generated new TSK 0
I20260809 00:53:26.099270 23649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:53:26.182884 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43919
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:26.576866 23669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:26.577265 23669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:26.577482 23669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:26.585486 23669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:26.585654 23669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:26.585855 23669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:53:26.597049 23669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:26.598826 23669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:26.600381 23669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:26.615783 23679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:26.615859 23677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:26.615828 23681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:26.617588 23669 server_base.cc:1034] running on GCE node
I20260809 00:53:26.618412 23669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:26.619586 23669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:26.620855 23669 hybrid_clock.cc:648] HybridClock initialized: now 1786236806620785 us; error 69 us; skew 500 ppm
I20260809 00:53:26.621236 23669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:26 dist-test-slave-j2vl krb5kdc[23494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236806, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:26.624833 23669 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:53:26.626801 23669 webserver.cc:460] Webserver started at http://127.9.225.1:44511/ using document root <none> and password file <none>
I20260809 00:53:26.627640 23669 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:26.627801 23669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:26.628158 23669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:26.631580 23669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "28e93e2550e8457c9cdb157ea9d0db6e"
format_stamp: "Formatted at 2026-08-09 00:53:26 on dist-test-slave-j2vl"
server_key: "de3c1ff7b67492589268afca3a5da8f7"
server_key_iv: "e3e809b1b57a76ce38d2e0b59dac6024"
server_key_version: "encryptionkey@0"
I20260809 00:53:26.632499 23669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "28e93e2550e8457c9cdb157ea9d0db6e"
format_stamp: "Formatted at 2026-08-09 00:53:26 on dist-test-slave-j2vl"
server_key: "de3c1ff7b67492589268afca3a5da8f7"
server_key_iv: "e3e809b1b57a76ce38d2e0b59dac6024"
server_key_version: "encryptionkey@0"
I20260809 00:53:26.639168 23669 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260809 00:53:26.643617 23691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:26.645274 23669 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260809 00:53:26.645598 23669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "28e93e2550e8457c9cdb157ea9d0db6e"
format_stamp: "Formatted at 2026-08-09 00:53:26 on dist-test-slave-j2vl"
server_key: "de3c1ff7b67492589268afca3a5da8f7"
server_key_iv: "e3e809b1b57a76ce38d2e0b59dac6024"
server_key_version: "encryptionkey@0"
I20260809 00:53:26.645821 23669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:26.670599 23669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:26.675738 23669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:26.676047 23669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:26.677156 23669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:26.678697 23669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:26.678784 23669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:26.678889 23669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:26.678936 23669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:26.715762 23669 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:46599
I20260809 00:53:26.715782 23866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:46599 every 8 connection(s)
I20260809 00:53:26.717301 23669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:53:26.718396 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 23669
I20260809 00:53:26.718609 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:53:26.719511 10116 external_mini_cluster.cc:1428] Setting key f41635dd9c5eb872b84285e0107782dd
Aug 09 00:53:26 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236806, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:53:26.744413 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:26.721720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:52561 (local address 127.9.225.62:43919)
0809 00:53:26.722265 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:26.722270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:26.723931 (+ 1661us) server_negotiation.cc:407] Connection header received
0809 00:53:26.726156 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:26.726165 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:26.726273 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:26.726404 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:26.728138 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:26.728916 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:26.730710 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:26.731154 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:26.731310 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:26.735933 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:26.735952 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:26.735960 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:26.736015 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:26.740240 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:26.741301 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:26.741310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:26.741316 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:26.741517 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:26.742096 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:26.742104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:26.742110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:26.742533 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:26.742725 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:26.743594 (+ 869us) server_negotiation.cc:299] Negotiation successful
0809 00:53:26.743884 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":311,"threads_started":1}
I20260809 00:53:26.745178 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:26.722494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:52561)
0809 00:53:26.723576 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:26.723679 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:26.725815 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:26.726692 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:26.726702 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:26.727307 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:26.727928 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:26.727960 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:26.729109 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:26.729113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:26.730535 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:26.730550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:26.731432 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:26.731441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:26.731725 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:26.732523 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:26.732549 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:26.735612 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:26.740488 (+ 4876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:26.740498 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:26.740516 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:26.741125 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:26.741670 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:26.741674 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:26.741676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:26.741937 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:26.742708 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:26.742716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:26.743139 (+ 423us) client_negotiation.cc:769] Sending connection context
0809 00:53:26.743403 (+ 264us) client_negotiation.cc:240] Negotiation successful
0809 00:53:26.743875 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":416,"threads_started":1}
I20260809 00:53:26.747659 23867 heartbeater.cc:347] Connected to a master server at 127.9.225.62:43919
I20260809 00:53:26.748078 23867 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:26.748931 23867 heartbeater.cc:511] Master 127.9.225.62:43919 requested a full tablet report, sending...
I20260809 00:53:26.751681 23568 ts_manager.cc:194] Registered new tserver with Master: 28e93e2550e8457c9cdb157ea9d0db6e (127.9.225.1:46599)
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
I20260809 00:53:26.753329 23568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:52561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:53:26.793685 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43919
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:27.184270 23875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:27.184615 23875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:27.184722 23875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:27.193019 23875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:27.193197 23875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:27.193333 23875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:53:27.204972 23875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:27.206688 23875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:27.208256 23875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:27.223518 23884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:27.223630 23883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:27.223608 23887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:27.224476 23875 server_base.cc:1034] running on GCE node
I20260809 00:53:27.225227 23875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:27.226401 23875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:27.227676 23875 hybrid_clock.cc:648] HybridClock initialized: now 1786236807227622 us; error 51 us; skew 500 ppm
I20260809 00:53:27.228085 23875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:27 dist-test-slave-j2vl krb5kdc[23494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236807, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:27.231751 23875 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:53:27.233644 23875 webserver.cc:460] Webserver started at http://127.9.225.2:42937/ using document root <none> and password file <none>
I20260809 00:53:27.234481 23875 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:27.234643 23875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:27.235013 23875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:27.238729 23875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "13b62338e4424c039b3d656a6791812b"
format_stamp: "Formatted at 2026-08-09 00:53:27 on dist-test-slave-j2vl"
server_key: "37d1cfb0818cf0dc98e9695e71d6141d"
server_key_iv: "2441efe2ba24bcd6f56233c0c08dc7c3"
server_key_version: "encryptionkey@0"
I20260809 00:53:27.239720 23875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "13b62338e4424c039b3d656a6791812b"
format_stamp: "Formatted at 2026-08-09 00:53:27 on dist-test-slave-j2vl"
server_key: "37d1cfb0818cf0dc98e9695e71d6141d"
server_key_iv: "2441efe2ba24bcd6f56233c0c08dc7c3"
server_key_version: "encryptionkey@0"
I20260809 00:53:27.246115 23875 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 00:53:27.250604 23899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:27.252331 23875 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:53:27.252564 23875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "13b62338e4424c039b3d656a6791812b"
format_stamp: "Formatted at 2026-08-09 00:53:27 on dist-test-slave-j2vl"
server_key: "37d1cfb0818cf0dc98e9695e71d6141d"
server_key_iv: "2441efe2ba24bcd6f56233c0c08dc7c3"
server_key_version: "encryptionkey@0"
I20260809 00:53:27.252806 23875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:27.272601 23875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:27.277890 23875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:27.278256 23875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:27.279422 23875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:27.281016 23875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:27.281096 23875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:27.281244 23875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:27.281301 23875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:27.321477 23875 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:36737
I20260809 00:53:27.321499 24086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:36737 every 8 connection(s)
I20260809 00:53:27.323559 23875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:53:27.327602 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 23875
I20260809 00:53:27.327795 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:53:27.328282 10116 external_mini_cluster.cc:1428] Setting key 1dfbe59aaba6daf6b2c343745bfc3e37
Aug 09 00:53:27 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236807, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:53:27.350320 24089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:27.326921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:38137 (local address 127.9.225.62:43919)
0809 00:53:27.327419 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:27.327425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:27.328783 (+ 1358us) server_negotiation.cc:407] Connection header received
0809 00:53:27.331084 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:27.331089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:27.331189 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:27.331315 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:27.332880 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:27.333703 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:27.335418 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:27.335884 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:27.336032 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:27.341962 (+ 5930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:27.341970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:27.341973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:27.342018 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:27.346119 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:27.347211 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:27.347220 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:27.347226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:27.347417 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:27.347991 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:27.347998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:27.348004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:27.348446 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:27.348611 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:27.349477 (+ 866us) server_negotiation.cc:299] Negotiation successful
0809 00:53:27.349774 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
I20260809 00:53:27.351488 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:27.327596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:38137)
0809 00:53:27.328573 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:27.328635 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:27.330740 (+ 2105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:27.331565 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:27.331574 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:27.332118 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:27.332697 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:27.332713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:27.333886 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:27.333891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:27.335239 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:27.335253 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:27.336174 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:27.336183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:27.336446 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:27.337920 (+ 1474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:27.337974 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:27.341597 (+ 3623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:27.346357 (+ 4760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:27.346367 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:27.346387 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:27.347045 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:27.347617 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:27.347621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:27.347623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:27.347863 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:27.348591 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:27.348598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:27.349008 (+ 410us) client_negotiation.cc:769] Sending connection context
0809 00:53:27.349284 (+ 276us) client_negotiation.cc:240] Negotiation successful
0809 00:53:27.349943 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":414,"threads_started":1}
I20260809 00:53:27.353408 24087 heartbeater.cc:347] Connected to a master server at 127.9.225.62:43919
I20260809 00:53:27.353788 24087 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:27.354681 24087 heartbeater.cc:511] Master 127.9.225.62:43919 requested a full tablet report, sending...
I20260809 00:53:27.356699 23568 ts_manager.cc:194] Registered new tserver with Master: 13b62338e4424c039b3d656a6791812b (127.9.225.2:36737)
I20260809 00:53:27.358088 23568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:38137
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:53:27.404894 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43919
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:53:27.757875 23867 heartbeater.cc:503] Master 127.9.225.62:43919 was elected leader, sending a full tablet report...
W20260809 00:53:27.797533 24095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:27.797955 24095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:27.798135 24095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:27.806113 24095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:27.806288 24095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:27.806488 24095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:53:27.817502 24095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:27.819064 24095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:27.820549 24095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:27.835630 24105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:27.835690 24104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:27.835717 24108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:27.836470 24095 server_base.cc:1034] running on GCE node
I20260809 00:53:27.837298 24095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:27.838864 24095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:27.840127 24095 hybrid_clock.cc:648] HybridClock initialized: now 1786236807840039 us; error 80 us; skew 500 ppm
I20260809 00:53:27.840546 24095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:27 dist-test-slave-j2vl krb5kdc[23494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236807, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:27.844246 24095 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:53:27.846208 24095 webserver.cc:460] Webserver started at http://127.9.225.3:46279/ using document root <none> and password file <none>
I20260809 00:53:27.847122 24095 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:27.847249 24095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:27.847618 24095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:27.851094 24095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "5f3fa37bac6a40658fabdce925c4094f"
format_stamp: "Formatted at 2026-08-09 00:53:27 on dist-test-slave-j2vl"
server_key: "975b8f89fb6a750597af2a086735fe81"
server_key_iv: "e6eeb614535cd61b0ce8eedd06b0596c"
server_key_version: "encryptionkey@0"
I20260809 00:53:27.852154 24095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "5f3fa37bac6a40658fabdce925c4094f"
format_stamp: "Formatted at 2026-08-09 00:53:27 on dist-test-slave-j2vl"
server_key: "975b8f89fb6a750597af2a086735fe81"
server_key_iv: "e6eeb614535cd61b0ce8eedd06b0596c"
server_key_version: "encryptionkey@0"
I20260809 00:53:27.858723 24095 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 00:53:27.863310 24120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:27.864991 24095 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:53:27.865173 24095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "5f3fa37bac6a40658fabdce925c4094f"
format_stamp: "Formatted at 2026-08-09 00:53:27 on dist-test-slave-j2vl"
server_key: "975b8f89fb6a750597af2a086735fe81"
server_key_iv: "e6eeb614535cd61b0ce8eedd06b0596c"
server_key_version: "encryptionkey@0"
I20260809 00:53:27.865520 24095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:27.887415 24095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:27.893751 24095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:27.894070 24095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:27.895565 24095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:27.897235 24095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:27.897326 24095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:27.897552 24095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:27.897611 24095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:27.935906 24095 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:40821
I20260809 00:53:27.936092 24295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:40821 every 8 connection(s)
I20260809 00:53:27.937531 24095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:53:27.938707 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 24095
I20260809 00:53:27.938894 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:53:27.939476 10116 external_mini_cluster.cc:1428] Setting key bd71a5a3d1405f2fbd8500224d1fd4ab
Aug 09 00:53:27 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236807, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:53:27.961762 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:27.940996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:33749 (local address 127.9.225.62:43919)
0809 00:53:27.941624 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:27.941631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:27.945062 (+ 3431us) server_negotiation.cc:407] Connection header received
0809 00:53:27.945126 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:27.945130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:27.945218 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:27.945335 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:27.947564 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:27.948423 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:27.950266 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:27.950750 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:27.950863 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:27.955267 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:27.955273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:27.955278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:27.955326 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:27.958306 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:27.959216 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:27.959220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:27.959223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:27.959368 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:27.959846 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:27.959849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:27.959851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:27.960155 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:27.960286 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:27.961160 (+ 874us) server_negotiation.cc:299] Negotiation successful
0809 00:53:27.961455 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":348,"threads_started":1}
I20260809 00:53:27.962769 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:27.941832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:33749)
0809 00:53:27.942834 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:27.942890 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:27.944697 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:27.945863 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:27.945874 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:27.946579 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:27.947313 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:27.947336 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:27.948595 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:27.948600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:27.950107 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:27.950124 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:27.951038 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:27.951044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:27.951272 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:27.952102 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:27.952135 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:27.954955 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:27.958495 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:27.958504 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:27.958524 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:27.959050 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:27.959528 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:27.959531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:27.959533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:27.959736 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:27.960269 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:27.960276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:27.960693 (+ 417us) client_negotiation.cc:769] Sending connection context
0809 00:53:27.960975 (+ 282us) client_negotiation.cc:240] Negotiation successful
0809 00:53:27.961517 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":365,"threads_started":1}
I20260809 00:53:27.964409 24297 heartbeater.cc:347] Connected to a master server at 127.9.225.62:43919
I20260809 00:53:27.964774 24297 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:27.965720 24297 heartbeater.cc:511] Master 127.9.225.62:43919 requested a full tablet report, sending...
I20260809 00:53:27.967968 23566 ts_manager.cc:194] Registered new tserver with Master: 5f3fa37bac6a40658fabdce925c4094f (127.9.225.3:40821)
I20260809 00:53:27.968999 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:53:27.969237 23566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:33749
WARNING: no policy specified for oryx/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab.
Entry for principal oryx/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab.
WARNING: no policy specified for oryx/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab.
Entry for principal oryx/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab.
WARNING: no policy specified for oryx/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab.
Entry for principal oryx/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab.
I20260809 00:53:28.181164 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 23669
Aug 09 00:53:28 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:53:28.200843 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:28.182738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.9.225.1:46599)
0809 00:53:28.183664 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:28.183672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:28.183704 (+ 32us) server_negotiation.cc:407] Connection header received
0809 00:53:28.183773 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:28.183780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:28.184027 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:28.184172 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:28.185819 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:28.186916 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:28.188290 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:28.188920 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:28.189040 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:28.194220 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:28.194242 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:28.194263 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:28.194312 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:28.197538 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:28.198499 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:28.198509 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:28.198512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:28.198668 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:28.199132 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:28.199137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:28.199143 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:28.199569 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:28.199849 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:28.200253 (+ 404us) server_negotiation.cc:299] Negotiation successful
0809 00:53:28.200522 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":551,"threads_started":1}
I20260809 00:53:28.202435 23859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:28.291177 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 23669
I20260809 00:53:28.320966 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 23875
Aug 09 00:53:28 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:53:28.341652 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:28.322053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.9.225.2:36737)
0809 00:53:28.322728 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:28.322736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:28.322763 (+ 27us) server_negotiation.cc:407] Connection header received
0809 00:53:28.322827 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:28.322833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:28.323056 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:28.323191 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:28.324996 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:28.326343 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:28.328206 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:28.328805 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:28.328917 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:28.334464 (+ 5547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:28.334483 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:28.334502 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:28.334549 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:28.338243 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:28.339174 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:28.339185 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:28.339187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:28.339353 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:28.339781 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:28.339791 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:28.339793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:28.340191 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:28.340537 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:28.340880 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 00:53:28.341184 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":361,"threads_started":1}
I20260809 00:53:28.343123 24079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:28.429581 24087 heartbeater.cc:503] Master 127.9.225.62:43919 was elected leader, sending a full tablet report...
I20260809 00:53:28.431356 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 23875
W20260809 00:53:28.466298 23550 connection.cc:535] server connection from 127.9.225.2:38137 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 00:53:28.466511 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 24095
Aug 09 00:53:28 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:53:28.489264 24324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:28.467839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.9.225.3:40821)
0809 00:53:28.468815 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:28.468828 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:28.468881 (+ 53us) server_negotiation.cc:407] Connection header received
0809 00:53:28.469017 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:28.469050 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:28.469634 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:28.470313 (+ 679us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:28.473167 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:28.475131 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:28.476586 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:28.477190 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:28.477316 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:28.482316 (+ 5000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:28.482334 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:28.482354 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:28.482408 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:28.485837 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:28.486689 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:28.486698 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:28.486701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:28.486858 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:28.487278 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:28.487282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:28.487285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:28.487711 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:28.488005 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:28.488354 (+ 349us) server_negotiation.cc:299] Negotiation successful
0809 00:53:28.488739 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":515,"threads_started":1}
I20260809 00:53:28.490826 24287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:28.579921 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 24095
I20260809 00:53:28.605270 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 23515
I20260809 00:53:28.606797 23616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:28.716068 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 23515
I20260809 00:53:28.752152 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43919
--webserver_interface=127.9.225.62
--webserver_port=36573
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:43919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:29.148591 24329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:29.148964 24329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:29.149137 24329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:29.157217 24329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:53:29.157451 24329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:29.157567 24329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:29.157658 24329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:53:29.157721 24329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:53:29.168834 24329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:43919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=36573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:29.170545 24329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:29.172075 24329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:29.186476 24343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:29.186637 24337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:29.186607 24340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:29.187592 24329 server_base.cc:1034] running on GCE node
I20260809 00:53:29.188627 24329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:29.190618 24329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:29.191941 24329 hybrid_clock.cc:648] HybridClock initialized: now 1786236809191891 us; error 76 us; skew 500 ppm
I20260809 00:53:29.192415 24329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:29 dist-test-slave-j2vl krb5kdc[23494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236809, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:29.201689 24329 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:53:29.203754 24329 webserver.cc:460] Webserver started at http://127.9.225.62:36573/ using document root <none> and password file <none>
I20260809 00:53:29.204733 24329 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:29.204924 24329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:29.211709 24329 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:53:29.215339 24351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:29.217167 24329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 00:53:29.217448 24329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "5057a020f0ee4412860f6c5c4bbe8461"
format_stamp: "Formatted at 2026-08-09 00:53:25 on dist-test-slave-j2vl"
server_key: "48cd0f43292cfc84e884dc3e4d4012e2"
server_key_iv: "ca79a51e649da1aa46add12d1da79908"
server_key_version: "encryptionkey@0"
I20260809 00:53:29.218658 24329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:29.240464 24329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:29.246326 24329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:29.246871 24329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:29.271873 24329 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:43919
I20260809 00:53:29.271914 24432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:43919 every 8 connection(s)
I20260809 00:53:29.273845 24329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:53:29.276580 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 24329
I20260809 00:53:29.277673 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:46599
--local_ip_for_outbound_sockets=127.9.225.1
--tserver_master_addrs=127.9.225.62:43919
--webserver_port=44511
--webserver_interface=127.9.225.1
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.225.1
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:53:29.282361 24434 sys_catalog.cc:263] Verifying existing consensus state
I20260809 00:53:29.286245 24434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461: Bootstrap starting.
I20260809 00:53:29.308326 24434 log.cc:824] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:29.319106 24434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 00:53:29.319782 24434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461: Bootstrap complete.
I20260809 00:53:29.330236 24434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5057a020f0ee4412860f6c5c4bbe8461" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43919 } }
I20260809 00:53:29.331291 24434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5057a020f0ee4412860f6c5c4bbe8461, State: Initialized, Role: FOLLOWER
I20260809 00:53:29.332043 24434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5057a020f0ee4412860f6c5c4bbe8461" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43919 } }
I20260809 00:53:29.332463 24434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:53:29.332640 24434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:53:29.332862 24434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 1 FOLLOWER]: Advancing to term 2
I20260809 00:53:29.337543 24434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5057a020f0ee4412860f6c5c4bbe8461" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43919 } }
I20260809 00:53:29.338325 24434 leader_election.cc:302] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5057a020f0ee4412860f6c5c4bbe8461; no voters:
I20260809 00:53:29.339068 24434 leader_election.cc:288] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 00:53:29.339228 24438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 00:53:29.340665 24438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [term 2 LEADER]: Becoming Leader. State: Replica: 5057a020f0ee4412860f6c5c4bbe8461, State: Running, Role: LEADER
I20260809 00:53:29.341415 24438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5057a020f0ee4412860f6c5c4bbe8461" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43919 } }
I20260809 00:53:29.341706 24434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:53:29.344839 24442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5057a020f0ee4412860f6c5c4bbe8461. Latest consensus state: current_term: 2 leader_uuid: "5057a020f0ee4412860f6c5c4bbe8461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5057a020f0ee4412860f6c5c4bbe8461" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43919 } } }
I20260809 00:53:29.345446 24442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:29.346514 24439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5057a020f0ee4412860f6c5c4bbe8461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5057a020f0ee4412860f6c5c4bbe8461" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43919 } } }
I20260809 00:53:29.346899 24439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461 [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:29.355556 24451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:53:29.362550 24451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:53:29.364200 24451 catalog_manager.cc:1259] Loaded cluster ID: a5cc9e49ec4746abb1eb130d0997d7fb
I20260809 00:53:29.364414 24451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:53:29.369230 24451 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:53:29.371186 24451 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5057a020f0ee4412860f6c5c4bbe8461: Loaded TSK: 0
I20260809 00:53:29.373095 24451 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 00:53:29.688510 24436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:29.688896 24436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:29.689069 24436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:29.697060 24436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:29.697268 24436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:29.697504 24436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:53:29.708817 24436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.1.keytab
--principal=oryx/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:46599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=44511
--tserver_master_addrs=127.9.225.62:43919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:29.710457 24436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:29.711974 24436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:29.729295 24474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:29.729476 24471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:29.730173 24436 server_base.cc:1034] running on GCE node
W20260809 00:53:29.730392 24477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:29.731204 24436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:29.732507 24436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:29.733764 24436 hybrid_clock.cc:648] HybridClock initialized: now 1786236809733718 us; error 69 us; skew 500 ppm
I20260809 00:53:29.734213 24436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:29 dist-test-slave-j2vl krb5kdc[23494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236809, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:29.738162 24436 init.cc:377] Logged in from keytab as oryx/127.9.225.1@KRBTEST.COM (short username oryx)
I20260809 00:53:29.740245 24436 webserver.cc:460] Webserver started at http://127.9.225.1:44511/ using document root <none> and password file <none>
I20260809 00:53:29.741084 24436 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:29.741268 24436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:29.747964 24436 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 00:53:29.751369 24489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:29.753124 24436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 00:53:29.753294 24436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "28e93e2550e8457c9cdb157ea9d0db6e"
format_stamp: "Formatted at 2026-08-09 00:53:26 on dist-test-slave-j2vl"
server_key: "de3c1ff7b67492589268afca3a5da8f7"
server_key_iv: "e3e809b1b57a76ce38d2e0b59dac6024"
server_key_version: "encryptionkey@0"
I20260809 00:53:29.754472 24436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:29.775043 24436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:29.780607 24436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:29.780889 24436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:29.782096 24436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:29.783823 24436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:29.783986 24436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:29.784143 24436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:29.784246 24436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:29.824136 24436 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:46599
I20260809 00:53:29.824193 24667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:46599 every 8 connection(s)
I20260809 00:53:29.825915 24436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:53:29.831823 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 24436
I20260809 00:53:29.832522 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:36737
--local_ip_for_outbound_sockets=127.9.225.2
--tserver_master_addrs=127.9.225.62:43919
--webserver_port=42937
--webserver_interface=127.9.225.2
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.225.2
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:53:29 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:29 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:29.848601 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:29.829517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:46389)
0809 00:53:29.830409 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:29.830469 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:29.832422 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:29.834333 (+ 1911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:29.834347 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:29.835093 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:29.835719 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:29.835739 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:29.837509 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:29.837523 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:29.838887 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:29.838906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:29.840135 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:29.840144 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:29.840443 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:29.841308 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:29.841403 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:29.847095 (+ 5692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":387,"threads_started":1}
W20260809 00:53:29.849148 24671 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:29.829313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:46389 (local address 127.9.225.62:43919)
0809 00:53:29.830137 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:29.830151 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:29.830615 (+ 464us) server_negotiation.cc:407] Connection header received
0809 00:53:29.833180 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:29.833194 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:29.833694 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:29.834133 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:29.835983 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:29.837261 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:29.839089 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:29.839802 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:29.840074 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:29.847722 (+ 7648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:46389: BlockingRecv error: recv got EOF from 127.9.225.1:46389 (error 108)
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":299,"threads_started":1}
W20260809 00:53:29.849525 24668 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:43919 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.225.62:43919: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Aug 09 00:53:29 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:29 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:29.861166 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:29.850439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:56277)
0809 00:53:29.850659 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:29.850679 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:29.850802 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:29.851194 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:29.851197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:29.851587 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:29.851752 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:29.851760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:29.853516 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:29.853521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:29.854731 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:29.854752 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:29.856032 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:29.856039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:29.856235 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:29.856982 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:29.857001 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:29.860908 (+ 3907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 00:53:29.861670 24671 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:29.850610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:56277 (local address 127.9.225.62:43919)
0809 00:53:29.850815 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:29.850823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:29.850852 (+ 29us) server_negotiation.cc:407] Connection header received
0809 00:53:29.850928 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:29.850936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:29.851029 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:29.851160 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:29.851931 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:29.853290 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:29.854941 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:29.855764 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:29.855970 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:29.861325 (+ 5355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:56277: BlockingRecv error: recv got EOF from 127.9.225.1:56277 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 00:53:29 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:29 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:29.872073 24672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:29.862174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:40629)
0809 00:53:29.862404 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:29.862428 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:29.862562 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:29.862991 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:29.862997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:29.863381 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:29.863563 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:29.863573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:29.864787 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:29.864792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:29.865944 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:29.865959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:29.866842 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:29.866849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:29.867087 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:29.867881 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:29.867903 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:29.871689 (+ 3786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260809 00:53:29.872375 24671 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:29.862320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:40629 (local address 127.9.225.62:43919)
0809 00:53:29.862531 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:29.862537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:29.862558 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:53:29.862744 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:29.862751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:29.862830 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:29.862953 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:29.863737 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:29.864584 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:29.866133 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:29.866648 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:29.866823 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:29.872080 (+ 5257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:40629: BlockingRecv error: recv got EOF from 127.9.225.1:40629 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260809 00:53:29.872579 24668 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 00:53:30.231257 24673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:30.231590 24673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:30.231760 24673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:30.239570 24673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:30.239714 24673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:30.239938 24673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:53:30.251127 24673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.2.keytab
--principal=oryx/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:36737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=42937
--tserver_master_addrs=127.9.225.62:43919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:30.252682 24673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:30.254282 24673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:30.270507 24681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:30.270440 24679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:30.271591 24673 server_base.cc:1034] running on GCE node
W20260809 00:53:30.270404 24678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:30.274241 24673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:30.275411 24673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:30.276713 24673 hybrid_clock.cc:648] HybridClock initialized: now 1786236810276649 us; error 62 us; skew 500 ppm
I20260809 00:53:30.277140 24673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236810, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:30.281148 24673 init.cc:377] Logged in from keytab as oryx/127.9.225.2@KRBTEST.COM (short username oryx)
I20260809 00:53:30.283145 24673 webserver.cc:460] Webserver started at http://127.9.225.2:42937/ using document root <none> and password file <none>
I20260809 00:53:30.283998 24673 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:30.284189 24673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:30.291198 24673 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 00:53:30.294548 24691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:30.296262 24673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 00:53:30.296504 24673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "13b62338e4424c039b3d656a6791812b"
format_stamp: "Formatted at 2026-08-09 00:53:27 on dist-test-slave-j2vl"
server_key: "37d1cfb0818cf0dc98e9695e71d6141d"
server_key_iv: "2441efe2ba24bcd6f56233c0c08dc7c3"
server_key_version: "encryptionkey@0"
I20260809 00:53:30.297720 24673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:30.328655 24673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:30.333969 24673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:30.334298 24673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:30.335496 24673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:30.337169 24673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:30.337308 24673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:30.337477 24673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:30.337571 24673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:30.374100 24673 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:36737
I20260809 00:53:30.374130 24875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:36737 every 8 connection(s)
I20260809 00:53:30.375727 24673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:53:30.381807 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 24673
I20260809 00:53:30.382490 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:40821
--local_ip_for_outbound_sockets=127.9.225.3
--tserver_master_addrs=127.9.225.62:43919
--webserver_port=46279
--webserver_interface=127.9.225.3
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.225.3
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 00:53:30.394186 24878 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:30.378622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:34027 (local address 127.9.225.62:43919)
0809 00:53:30.379134 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:30.379138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:30.380028 (+ 890us) server_negotiation.cc:407] Connection header received
0809 00:53:30.382275 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:30.382281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:30.382386 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:30.382510 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:30.384236 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.384959 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.386749 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.387190 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.387335 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.393812 (+ 6477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:34027: BlockingRecv error: recv got EOF from 127.9.225.2:34027 (error 108)
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":287,"threads_started":1}
I20260809 00:53:30.394783 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:30.379091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:34027)
0809 00:53:30.379846 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:30.379898 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:30.381912 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:30.382791 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:30.382801 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:30.383370 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:30.384015 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.384035 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.385151 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.385155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.386570 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.386585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.387462 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.387469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.387726 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.388492 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:30.388519 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:30.393490 (+ 4971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":304,"threads_started":1}
W20260809 00:53:30.395638 24876 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:43919 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.225.62:43919: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:30.407311 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:30.396334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:55629)
0809 00:53:30.396602 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:30.396625 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:30.396752 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:30.397110 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:30.397117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:30.397691 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:30.397947 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.397962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.399035 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.399043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.400746 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.400768 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.401715 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.401723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.401977 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.402700 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:30.402729 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:30.406889 (+ 4160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260809 00:53:30.407426 24878 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:30.396500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:55629 (local address 127.9.225.62:43919)
0809 00:53:30.396682 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:30.396686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:30.396706 (+ 20us) server_negotiation.cc:407] Connection header received
0809 00:53:30.396885 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:30.396889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:30.396958 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:30.397044 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:30.398132 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.398870 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.400944 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.401510 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.401678 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.407177 (+ 5499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:55629: BlockingRecv error: recv got EOF from 127.9.225.2:55629 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:30.417440 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:30.408442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:46455)
0809 00:53:30.408639 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:30.408661 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:30.408787 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:30.409151 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:30.409156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:30.409571 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:30.409741 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.409750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.410828 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.410835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.411783 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.411794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.412522 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.412528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.412716 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.413416 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:30.413437 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:30.417066 (+ 3629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 00:53:30.417835 24876 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 00:53:30.417994 24878 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:30.408538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:46455 (local address 127.9.225.62:43919)
0809 00:53:30.408780 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:30.408787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:30.408807 (+ 20us) server_negotiation.cc:407] Connection header received
0809 00:53:30.408939 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:30.408943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:30.409006 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:30.409116 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:30.409903 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.410673 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.411938 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.412360 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.412505 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.417436 (+ 4931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:46455: BlockingRecv error: recv got EOF from 127.9.225.2:46455 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260809 00:53:30.775403 24883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:30.775766 24883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:30.775872 24883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:30.783849 24883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:30.783985 24883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:30.784106 24883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:53:30.795038 24883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/krb5kdc/oryx_127.9.225.3.keytab
--principal=oryx/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:40821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=46279
--tserver_master_addrs=127.9.225.62:43919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:30.796612 24883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:30.798189 24883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:30.814677 24890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:30.814702 24893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:30.814644 24889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:30.815697 24883 server_base.cc:1034] running on GCE node
I20260809 00:53:30.816504 24883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:30.817806 24883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:30.819074 24883 hybrid_clock.cc:648] HybridClock initialized: now 1786236810818999 us; error 77 us; skew 500 ppm
I20260809 00:53:30.819486 24883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236810, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:30.823470 24883 init.cc:377] Logged in from keytab as oryx/127.9.225.3@KRBTEST.COM (short username oryx)
I20260809 00:53:30.829854 24883 webserver.cc:460] Webserver started at http://127.9.225.3:46279/ using document root <none> and password file <none>
I20260809 00:53:30.830717 24883 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:30.830905 24883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:30.838408 24883 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 00:53:30.842175 24902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:30.844058 24883 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:53:30.844318 24883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "5f3fa37bac6a40658fabdce925c4094f"
format_stamp: "Formatted at 2026-08-09 00:53:27 on dist-test-slave-j2vl"
server_key: "975b8f89fb6a750597af2a086735fe81"
server_key_iv: "e6eeb614535cd61b0ce8eedd06b0596c"
server_key_version: "encryptionkey@0"
I20260809 00:53:30.845546 24883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:30.870234 24883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:30.875986 24883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:30.876328 24883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:30.877641 24883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:30.879432 24883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:30.879565 24883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:30.879698 24883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:30.879801 24883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:30.884035 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:30.873412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:40111)
0809 00:53:30.874026 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:30.874048 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:30.874222 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:30.874614 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:30.874619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:30.875148 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:30.875358 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.875367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.876558 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.876564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.877895 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.877911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.878837 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.878848 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.879061 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.879808 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:30.879837 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:30.883679 (+ 3842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":300,"threads_started":1}
W20260809 00:53:30.884315 24878 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:30.873711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:40111 (local address 127.9.225.62:43919)
0809 00:53:30.873932 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:30.873938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:30.874146 (+ 208us) server_negotiation.cc:407] Connection header received
0809 00:53:30.874375 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:30.874385 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:30.874464 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:30.874577 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:30.875534 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.876371 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.878065 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.878538 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.878729 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.884027 (+ 5298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:40111: BlockingRecv error: recv got EOF from 127.9.225.1:40111 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 00:53:30.916996 24883 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:40821
I20260809 00:53:30.917083 25071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:40821 every 8 connection(s)
I20260809 00:53:30.918751 24883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:53:30.920869 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 24883
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 00:53:30.938917 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:30.922306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:57927 (local address 127.9.225.62:43919)
0809 00:53:30.922960 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:30.922965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:30.924130 (+ 1165us) server_negotiation.cc:407] Connection header received
0809 00:53:30.926974 (+ 2844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:30.926981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:30.927059 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:30.927177 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:30.928953 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.929934 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.931836 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.932496 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.932649 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.938599 (+ 5950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:57927: BlockingRecv error: recv got EOF from 127.9.225.3:57927 (error 108)
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":522,"threads_started":1}
I20260809 00:53:30.939659 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:30.922660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:57927)
0809 00:53:30.923844 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:30.923928 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:30.926672 (+ 2744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:30.927450 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:30.927460 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:30.928068 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:30.928723 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.928743 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.930112 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.930119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.931635 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.931651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.932804 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.932814 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.933079 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.933826 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:30.933857 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:30.938096 (+ 4239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":450,"threads_started":1}
W20260809 00:53:30.940598 25073 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:43919 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.225.62:43919: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
I20260809 00:53:30.943053 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:30.922200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.9.225.62:43919)
0809 00:53:30.922786 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:30.922800 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:30.925992 (+ 3192us) server_negotiation.cc:407] Connection header received
0809 00:53:30.926421 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:30.926426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:30.926519 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:30.926635 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:30.928719 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.929648 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.931011 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.931441 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.931594 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.934727 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:30.934737 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:30.934759 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:30.934800 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:30.938363 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:30.939444 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:30.939452 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:30.939527 (+ 75us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:30.939784 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:30.940520 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:30.940524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:30.940526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:30.941182 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:30.941582 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:30.942286 (+ 704us) server_negotiation.cc:299] Negotiation successful
0809 00:53:30.942735 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366}
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:30.951028 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:30.941313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:39343)
0809 00:53:30.941634 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:30.941656 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:30.941776 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:30.942145 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:30.942151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:30.942523 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:30.942758 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.942773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.944057 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.944063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.945631 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.945651 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.946593 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.946599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.946834 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.947520 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:30.947545 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:30.950787 (+ 3242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191}
W20260809 00:53:30.951426 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:30.941521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:39343 (local address 127.9.225.62:43919)
0809 00:53:30.941803 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:30.941810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:30.941832 (+ 22us) server_negotiation.cc:407] Connection header received
0809 00:53:30.941885 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:30.941890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:30.941957 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:30.942064 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:30.942924 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.943891 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.945805 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.946415 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.946569 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.951097 (+ 4528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:39343: BlockingRecv error: recv got EOF from 127.9.225.3:39343 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:30 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:30.961253 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:30.951799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:48773)
0809 00:53:30.951997 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:30.952023 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:30.952121 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:30.952419 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:30.952423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:30.952732 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:30.952897 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.952936 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.954187 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.954192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.955256 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:30.955274 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.956100 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.956105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:30.956241 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.956934 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:30.956953 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:30.960908 (+ 3955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260809 00:53:30.961455 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:30.951927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:48773 (local address 127.9.225.62:43919)
0809 00:53:30.952117 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:30.952121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:30.952141 (+ 20us) server_negotiation.cc:407] Connection header received
0809 00:53:30.952201 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:30.952205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:30.952282 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:30.952374 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:30.953138 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.954008 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.955414 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:30.955954 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:30.956056 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:30.961134 (+ 5078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:48773: BlockingRecv error: recv got EOF from 127.9.225.3:48773 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260809 00:53:30.961834 25073 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 00:53:31 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:31 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:31.428328 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:31.418547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:39043)
0809 00:53:31.419135 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:31.419158 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:31.419294 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:31.419651 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:31.419655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:31.420313 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:31.420605 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:31.420621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:31.421943 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:31.421950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:31.423101 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:31.423115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:31.423943 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:31.423948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:31.424099 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:31.424857 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:31.424877 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:31.428074 (+ 3197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":315,"threads_started":1}
W20260809 00:53:31.428624 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:31.418790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:39043 (local address 127.9.225.62:43919)
0809 00:53:31.419043 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:31.419050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:31.419244 (+ 194us) server_negotiation.cc:407] Connection header received
0809 00:53:31.419408 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:31.419412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:31.419509 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:31.419599 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:31.420766 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:31.421776 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:31.423247 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:31.423748 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:31.423851 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:31.428366 (+ 4515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:39043: BlockingRecv error: recv got EOF from 127.9.225.2:39043 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 09 00:53:31 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:31 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:31.894387 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:31.885173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:38437)
0809 00:53:31.885788 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:31.885953 (+ 165us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:31.886095 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:31.886427 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:31.886431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:31.886851 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:31.887020 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:31.887029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:31.888090 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:31.888094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:31.889027 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:31.889040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:31.889783 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:31.889788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:31.889906 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:31.890615 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:31.890634 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:31.893994 (+ 3360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":392,"threads_started":1}
W20260809 00:53:31.894564 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:31.885259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:38437 (local address 127.9.225.62:43919)
0809 00:53:31.885564 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:31.885575 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:31.886060 (+ 485us) server_negotiation.cc:407] Connection header received
0809 00:53:31.886205 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:31.886209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:31.886294 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:31.886383 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:31.887175 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:31.887951 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:31.889154 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:31.889639 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:31.889732 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:31.894292 (+ 4560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:38437: BlockingRecv error: recv got EOF from 127.9.225.1:38437 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Aug 09 00:53:31 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:31 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:31.972059 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:31.962512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:59507)
0809 00:53:31.963049 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:31.963066 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:31.963191 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:31.963595 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:31.963599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:31.964146 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:31.964356 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:31.964366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:31.965528 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:31.965532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:31.966585 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:31.966599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:31.967390 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:31.967395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:31.967543 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:31.968186 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:31.968205 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:31.971704 (+ 3499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":289,"threads_started":1}
W20260809 00:53:31.972261 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:31.962610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:59507 (local address 127.9.225.62:43919)
0809 00:53:31.962888 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:31.962895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:31.963153 (+ 258us) server_negotiation.cc:407] Connection header received
0809 00:53:31.963342 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:31.963346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:31.963417 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:31.963517 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:31.964531 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:31.965313 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:31.966724 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:31.967183 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:31.967293 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:31.972019 (+ 4726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:59507: BlockingRecv error: recv got EOF from 127.9.225.3:59507 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 09 00:53:32 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:32 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:32.439147 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:32.429546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:49213)
0809 00:53:32.430163 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:32.430185 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:32.430319 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:32.430701 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:32.430705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:32.431135 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:32.431292 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:32.431300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:32.432719 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:32.432724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:32.433786 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:32.433801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:32.434633 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:32.434639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:32.434842 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:32.435542 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:32.435563 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:32.438776 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":354,"threads_started":1}
W20260809 00:53:32.439564 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:32.429652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:49213 (local address 127.9.225.62:43919)
0809 00:53:32.429848 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:32.429852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:32.430273 (+ 421us) server_negotiation.cc:407] Connection header received
0809 00:53:32.430446 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:32.430450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:32.430521 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:32.430615 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:32.431479 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:32.432541 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:32.433960 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:32.434451 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:32.434606 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:32.439129 (+ 4523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:49213: BlockingRecv error: recv got EOF from 127.9.225.2:49213 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 09 00:53:32 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:32 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:32.905455 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:32.895828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:50867)
0809 00:53:32.896417 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:32.896440 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:32.896580 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:32.897030 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:32.897038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:32.897544 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:32.897831 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:32.897843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:32.899236 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:32.899242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:32.900192 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:32.900205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:32.901055 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:32.901060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:32.901205 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:32.902035 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:32.902055 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:32.905095 (+ 3040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":345,"threads_started":1}
W20260809 00:53:32.905646 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:32.896091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:50867 (local address 127.9.225.62:43919)
0809 00:53:32.896387 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:32.896394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:32.896549 (+ 155us) server_negotiation.cc:407] Connection header received
0809 00:53:32.896720 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:32.896725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:32.896832 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:32.896951 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:32.898028 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:32.899070 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:32.900365 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:32.900841 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:32.900948 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:32.905415 (+ 4467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:50867: BlockingRecv error: recv got EOF from 127.9.225.1:50867 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 09 00:53:32 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:32 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:32.982270 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:32.973244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:48707)
0809 00:53:32.973913 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:32.973951 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:32.974084 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:32.974702 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:32.974707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:32.975197 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:32.975356 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:32.975365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:32.976419 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:32.976424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:32.977360 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:32.977373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:32.978068 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:32.978072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:32.978192 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:32.978880 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:32.978900 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:32.982007 (+ 3107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":413,"threads_started":1}
W20260809 00:53:32.982527 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:32.973301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:48707 (local address 127.9.225.62:43919)
0809 00:53:32.973563 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:32.973570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:32.974031 (+ 461us) server_negotiation.cc:407] Connection header received
0809 00:53:32.974240 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:32.974244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:32.974313 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:32.974420 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:32.975512 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:32.976296 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:32.977495 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:32.977922 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:32.978029 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:32.982266 (+ 4237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:48707: BlockingRecv error: recv got EOF from 127.9.225.3:48707 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Aug 09 00:53:33 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:33 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:33.450309 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:33.440501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:51543)
0809 00:53:33.441093 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:33.441125 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:33.441258 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:33.441687 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:33.441690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:33.442193 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:33.442391 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:33.442400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:33.443642 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:33.443646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:33.444610 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:33.444623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:33.445482 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:33.445488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:33.445645 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:33.446359 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:33.446383 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:33.450005 (+ 3622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":347,"threads_started":1}
W20260809 00:53:33.450531 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:33.440648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:51543 (local address 127.9.225.62:43919)
0809 00:53:33.440881 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:33.440886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:33.441207 (+ 321us) server_negotiation.cc:407] Connection header received
0809 00:53:33.441452 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:33.441457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:33.441530 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:33.441626 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:33.442574 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:33.443490 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:33.444765 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:33.445227 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:33.445333 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:33.450276 (+ 4943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:51543: BlockingRecv error: recv got EOF from 127.9.225.2:51543 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 09 00:53:33 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:33 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:33.919124 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:33.906724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:60685)
0809 00:53:33.907298 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:33.907315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:33.907502 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:33.907925 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:33.907929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:33.908391 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:33.908556 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:33.908566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:33.910128 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:33.910137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:33.911713 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:33.911732 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:33.912727 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:33.912733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:33.912951 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:33.913815 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:33.913844 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:33.918764 (+ 4920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":279,"threads_started":1}
W20260809 00:53:33.919422 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:33.906884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:60685 (local address 127.9.225.62:43919)
0809 00:53:33.907125 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:33.907131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:33.907413 (+ 282us) server_negotiation.cc:407] Connection header received
0809 00:53:33.907642 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:33.907650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:33.907756 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:33.907866 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:33.908719 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:33.909937 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:33.911887 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:33.912362 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:33.912719 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:33.919121 (+ 6402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:60685: BlockingRecv error: recv got EOF from 127.9.225.1:60685 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 00:53:33 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:33 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:33.992612 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:33.983566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:42057)
0809 00:53:33.984165 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:33.984196 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:33.984339 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:33.984733 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:33.984739 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:33.985259 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:33.985521 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:33.985532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:33.986731 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:33.986739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:33.987655 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:33.987667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:33.988394 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:33.988400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:33.988578 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:33.989202 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:33.989219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:33.992324 (+ 3105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":348,"threads_started":1}
W20260809 00:53:33.992787 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:33.983722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:42057 (local address 127.9.225.62:43919)
0809 00:53:33.983904 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:33.983908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:33.984313 (+ 405us) server_negotiation.cc:407] Connection header received
0809 00:53:33.984504 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:33.984509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:33.984583 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:33.984700 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:33.985696 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:33.986585 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:33.987805 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:33.988221 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:33.988364 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:33.992576 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:42057: BlockingRecv error: recv got EOF from 127.9.225.3:42057 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 00:53:34 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:34 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:34.466619 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:34.451439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:39411)
0809 00:53:34.451988 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:34.452008 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:34.452186 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:34.452539 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:34.452544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:34.452992 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:34.453155 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:34.453165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:34.454352 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:34.454360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:34.455437 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:34.455451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:34.456185 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:34.456192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:34.456400 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:34.461805 (+ 5405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:34.461838 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:34.466264 (+ 4426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":324,"threads_started":1}
W20260809 00:53:34.466950 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:34.451502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:39411 (local address 127.9.225.62:43919)
0809 00:53:34.451733 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:34.451740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:34.452112 (+ 372us) server_negotiation.cc:407] Connection header received
0809 00:53:34.452318 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:34.452330 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:34.452397 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:34.452503 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:34.453322 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:34.454200 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:34.455589 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:34.456021 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:34.456165 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:34.466591 (+ 10426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:39411: BlockingRecv error: recv got EOF from 127.9.225.2:39411 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 09 00:53:34 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:34 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:34.929970 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:34.920549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:40257)
0809 00:53:34.921716 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:34.921733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:34.921890 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:34.922298 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:34.922304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:34.922864 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:34.923122 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:34.923137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:34.924270 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:34.924274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:34.925268 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:34.925281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:34.926111 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:34.926116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:34.926242 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:34.926842 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:34.926860 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:34.929686 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":527,"threads_started":1}
W20260809 00:53:34.930167 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:34.920655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:40257 (local address 127.9.225.62:43919)
0809 00:53:34.920898 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:34.920906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:34.921824 (+ 918us) server_negotiation.cc:407] Connection header received
0809 00:53:34.922042 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:34.922048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:34.922131 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:34.922233 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:34.923279 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:34.924148 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:34.925473 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:34.925965 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:34.926064 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:34.929949 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:40257: BlockingRecv error: recv got EOF from 127.9.225.1:40257 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 00:53:34 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:34 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:35.003299 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:34.993930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:58463)
0809 00:53:34.994516 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:34.994536 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:34.994733 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:34.995125 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:34.995130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:34.995607 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:34.995775 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:34.995785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:34.996852 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:34.996858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:34.998011 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:34.998025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:34.998810 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:34.998816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:34.999127 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:34.999761 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:34.999780 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:35.002993 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":323,"threads_started":1}
W20260809 00:53:35.003767 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:34.994015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:58463 (local address 127.9.225.62:43919)
0809 00:53:34.994327 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:34.994334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:34.994632 (+ 298us) server_negotiation.cc:407] Connection header received
0809 00:53:34.994887 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:34.994892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:34.994964 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:34.995078 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:34.995945 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:34.996696 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:34.998163 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:34.998650 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:34.998794 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:35.003492 (+ 4698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:58463: BlockingRecv error: recv got EOF from 127.9.225.3:58463 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Aug 09 00:53:35 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:35 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:35.479741 25100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:35.468115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:53119)
0809 00:53:35.469018 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:35.469038 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:35.469222 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:35.469677 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:35.469684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:35.470133 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:35.470313 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:35.470323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:35.471970 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:35.471975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:35.473067 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:35.473081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:35.473954 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:35.473960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:35.474150 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:35.474881 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:35.474901 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:35.479291 (+ 4390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":390,"threads_started":1}
W20260809 00:53:35.479795 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:35.468617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:53119 (local address 127.9.225.62:43919)
0809 00:53:35.468817 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:35.468824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:35.469130 (+ 306us) server_negotiation.cc:407] Connection header received
0809 00:53:35.469389 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:35.469395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:35.469502 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:35.469629 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:35.470495 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:35.471806 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:35.473241 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:35.473776 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:35.473931 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:35.479605 (+ 5674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:53119: BlockingRecv error: recv got EOF from 127.9.225.2:53119 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 00:53:35 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:35 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:35.940030 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:35.931132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:46589)
0809 00:53:35.931710 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:35.931727 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:35.931920 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:35.932273 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:35.932277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:35.932711 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:35.932885 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:35.932894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:35.934036 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:35.934043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:35.935007 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:35.935019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:35.935807 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:35.935824 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:35.936007 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:35.936626 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:35.936645 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:35.939678 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":324,"threads_started":1}
W20260809 00:53:35.940207 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:35.931212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:46589 (local address 127.9.225.62:43919)
0809 00:53:35.931423 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:35.931429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:35.931824 (+ 395us) server_negotiation.cc:407] Connection header received
0809 00:53:35.932054 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:35.932065 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:35.932130 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:35.932238 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:35.933034 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:35.933890 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:35.935174 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:35.935631 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:35.935767 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:35.939934 (+ 4167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:46589: BlockingRecv error: recv got EOF from 127.9.225.1:46589 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 00:53:36 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:36 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:36.014026 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:36.004435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:55027)
0809 00:53:36.004994 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:36.005013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:36.005200 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:36.005658 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:36.005663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:36.006126 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:36.006308 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:36.006319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:36.007418 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:36.007424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:36.008353 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:36.008365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:36.009098 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:36.009104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:36.009296 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:36.010070 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:36.010091 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:36.013633 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":288,"threads_started":1}
W20260809 00:53:36.014160 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:36.004667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:55027 (local address 127.9.225.62:43919)
0809 00:53:36.004924 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:36.004932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:36.005167 (+ 235us) server_negotiation.cc:407] Connection header received
0809 00:53:36.005404 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:36.005416 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:36.005492 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:36.005617 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:36.006478 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:36.007264 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:36.008507 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:36.008933 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:36.009069 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:36.013907 (+ 4838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:55027: BlockingRecv error: recv got EOF from 127.9.225.3:55027 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 09 00:53:36 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:36 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:36.490309 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:36.480971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:43109)
0809 00:53:36.481603 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:36.481625 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:36.481806 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:36.482178 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:36.482183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:36.482633 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:36.482805 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:36.482814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:36.483878 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:36.483888 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:36.484806 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:36.484818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:36.485669 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:36.485675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:36.485876 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:36.486613 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:36.486632 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:36.489972 (+ 3340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":376,"threads_started":1}
W20260809 00:53:36.490484 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:36.481102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:43109 (local address 127.9.225.62:43919)
0809 00:53:36.481400 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:36.481408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:36.481718 (+ 310us) server_negotiation.cc:407] Connection header received
0809 00:53:36.481937 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:36.481943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:36.482029 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:36.482139 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:36.482958 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:36.483704 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:36.484995 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:36.485496 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:36.485637 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:36.490226 (+ 4589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:43109: BlockingRecv error: recv got EOF from 127.9.225.2:43109 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 09 00:53:36 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:36 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:36.951215 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:36.941133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:51549)
0809 00:53:36.941861 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:36.941879 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:36.942001 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:36.942446 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:36.942450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:36.943017 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:36.943207 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:36.943217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:36.944391 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:36.944396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:36.945486 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:36.945502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:36.946368 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:36.946374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:36.946509 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:36.947233 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:36.947268 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:36.950810 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":347,"threads_started":1}
W20260809 00:53:36.951439 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:36.941265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:51549 (local address 127.9.225.62:43919)
0809 00:53:36.941582 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:36.941588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:36.941964 (+ 376us) server_negotiation.cc:407] Connection header received
0809 00:53:36.942184 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:36.942188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:36.942266 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:36.942368 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:36.943391 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:36.944227 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:36.945655 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:36.946188 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:36.946303 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:36.951164 (+ 4861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:51549: BlockingRecv error: recv got EOF from 127.9.225.1:51549 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 09 00:53:37 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:37 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:37.026204 25112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:37.015385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:33035)
0809 00:53:37.016181 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:37.016227 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:37.016399 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:37.016853 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:37.016859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:37.017476 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:37.017662 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:37.017673 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:37.018950 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:37.018956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:37.020133 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:37.020169 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:37.021303 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:37.021311 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:37.021572 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:37.022321 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:37.022349 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:37.025956 (+ 3607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":493,"threads_started":1}
W20260809 00:53:37.026682 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:37.015591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:33035 (local address 127.9.225.62:43919)
0809 00:53:37.016008 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:37.016014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:37.016356 (+ 342us) server_negotiation.cc:407] Connection header received
0809 00:53:37.016572 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:37.016578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:37.016675 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:37.016803 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:37.017904 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:37.018774 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:37.020330 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:37.021040 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:37.021206 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:37.026226 (+ 5020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:33035: BlockingRecv error: recv got EOF from 127.9.225.3:33035 (error 108)
Metrics: {"server-negotiator.queue_time_us":237}
Aug 09 00:53:37 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:37 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:37.500977 25113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:37.491545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:37089)
0809 00:53:37.492310 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:37.492337 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:37.492503 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:37.492954 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:37.492960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:37.493623 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:37.493827 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:37.493838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:37.495001 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:37.495006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:37.495976 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:37.495989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:37.496680 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:37.496685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:37.496808 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:37.497583 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:37.497607 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:37.500668 (+ 3061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":367,"threads_started":1}
W20260809 00:53:37.501258 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:37.491589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:37089 (local address 127.9.225.62:43919)
0809 00:53:37.491843 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:37.491848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:37.492445 (+ 597us) server_negotiation.cc:407] Connection header received
0809 00:53:37.492662 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:37.492671 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:37.492778 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:37.492889 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:37.494045 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:37.494826 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:37.496117 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:37.496526 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:37.496623 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:37.500945 (+ 4322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:37089: BlockingRecv error: recv got EOF from 127.9.225.2:37089 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 09 00:53:37 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:37 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:37.963099 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:37.952593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:55397)
0809 00:53:37.953225 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:37.953242 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:37.953460 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:37.953932 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:37.953936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:37.954415 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:37.954583 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:37.954593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:37.956044 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:37.956051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:37.957039 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:37.957054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:37.958380 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:37.958388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:37.958640 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:37.959383 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:37.959403 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:37.962796 (+ 3393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":340,"threads_started":1}
W20260809 00:53:37.963352 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:37.952792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:55397 (local address 127.9.225.62:43919)
0809 00:53:37.953049 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:37.953056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:37.953369 (+ 313us) server_negotiation.cc:407] Connection header received
0809 00:53:37.953640 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:37.953663 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:37.953776 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:37.953899 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:37.954768 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:37.955872 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:37.957641 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:37.958142 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:37.958295 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:37.963079 (+ 4784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:55397: BlockingRecv error: recv got EOF from 127.9.225.1:55397 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 09 00:53:38 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:38 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:38.036976 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:38.027509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:44713)
0809 00:53:38.028158 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:38.028187 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:38.028328 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:38.028668 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:38.028672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:38.029143 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:38.029325 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:38.029334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:38.030491 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:38.030496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:38.031526 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:38.031545 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:38.032619 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:38.032626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:38.032816 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:38.033528 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:38.033546 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:38.036695 (+ 3149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":312,"threads_started":1}
W20260809 00:53:38.037156 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:38.027695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:44713 (local address 127.9.225.62:43919)
0809 00:53:38.027891 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:38.027896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:38.028312 (+ 416us) server_negotiation.cc:407] Connection header received
0809 00:53:38.028437 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:38.028441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:38.028515 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:38.028625 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:38.029572 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:38.030337 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:38.031696 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:38.032433 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:38.032585 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:38.036944 (+ 4359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:44713: BlockingRecv error: recv got EOF from 127.9.225.3:44713 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 09 00:53:38 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:38 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:38.511902 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:38.502117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:37077)
0809 00:53:38.502795 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:38.502816 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:38.503014 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:38.503393 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:38.503400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:38.503865 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:38.504025 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:38.504034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:38.505285 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:38.505293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:38.506247 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:38.506259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:38.507273 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:38.507280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:38.507495 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:38.508232 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:38.508251 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:38.511653 (+ 3402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":356,"threads_started":1}
W20260809 00:53:38.512192 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:38.502303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:37077 (local address 127.9.225.62:43919)
0809 00:53:38.502635 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:38.502641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:38.502943 (+ 302us) server_negotiation.cc:407] Connection header received
0809 00:53:38.503157 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:38.503167 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:38.503235 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:38.503348 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:38.504183 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:38.505121 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:38.506412 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:38.507011 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:38.507215 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:38.511903 (+ 4688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:37077: BlockingRecv error: recv got EOF from 127.9.225.2:37077 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 09 00:53:38 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:38 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:38.973857 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:38.964282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:42403)
0809 00:53:38.964860 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:38.964877 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:38.965000 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:38.965299 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:38.965302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:38.965760 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:38.965997 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:38.966008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:38.967017 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:38.967021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:38.968000 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:38.968014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:38.968786 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:38.968792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:38.968938 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:38.969705 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:38.969729 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:38.973522 (+ 3793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":311,"threads_started":1}
W20260809 00:53:38.974200 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:38.964358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:42403 (local address 127.9.225.62:43919)
0809 00:53:38.964617 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:38.964621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:38.964962 (+ 341us) server_negotiation.cc:407] Connection header received
0809 00:53:38.965099 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:38.965103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:38.965169 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:38.965250 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:38.966145 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:38.966876 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:38.968173 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:38.968619 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:38.968728 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:38.973872 (+ 5144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:42403: BlockingRecv error: recv got EOF from 127.9.225.1:42403 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 09 00:53:39 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:39 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:39.047823 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:39.038500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:45271)
0809 00:53:39.039132 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:39.039150 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:39.039273 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:39.039708 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:39.039715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:39.040163 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:39.040397 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:39.040408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:39.041544 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:39.041550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:39.042572 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:39.042586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:39.043392 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:39.043398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:39.043595 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:39.044254 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:39.044271 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:39.047441 (+ 3170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":341,"threads_started":1}
W20260809 00:53:39.047995 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:39.038614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:45271 (local address 127.9.225.62:43919)
0809 00:53:39.038855 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:39.038868 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:39.039219 (+ 351us) server_negotiation.cc:407] Connection header received
0809 00:53:39.039417 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:39.039422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:39.039531 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:39.039643 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:39.040559 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:39.041340 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:39.042718 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:39.043186 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:39.043332 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:39.047694 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:45271: BlockingRecv error: recv got EOF from 127.9.225.3:45271 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 00:53:39 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:39 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:39.527470 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:39.513223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:48017)
0809 00:53:39.514185 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:39.514207 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:39.514408 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:39.514827 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:39.514832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:39.515306 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:39.515487 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:39.515497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:39.516667 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:39.516673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:39.517992 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:39.518013 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:39.518901 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:39.518906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:39.519059 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:39.523239 (+ 4180us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:39.523266 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:39.527048 (+ 3782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":392,"threads_started":1}
W20260809 00:53:39.527601 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:39.513398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:48017 (local address 127.9.225.62:43919)
0809 00:53:39.513974 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:39.513985 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:39.514316 (+ 331us) server_negotiation.cc:407] Connection header received
0809 00:53:39.514552 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:39.514559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:39.514649 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:39.514786 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:39.515659 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:39.516496 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:39.518165 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:39.518684 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:39.518816 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:39.527334 (+ 8518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:48017: BlockingRecv error: recv got EOF from 127.9.225.2:48017 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 09 00:53:39 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:39 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:39.984928 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:39.975181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:34973)
0809 00:53:39.975823 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:39.975841 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:39.976023 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:39.976402 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:39.976407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:39.976939 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:39.977098 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:39.977107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:39.978516 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:39.978523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:39.979512 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:39.979524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:39.980319 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:39.980326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:39.980527 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:39.981289 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:39.981309 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:39.984655 (+ 3346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":382,"threads_started":1}
W20260809 00:53:39.985234 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:39.975377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:34973 (local address 127.9.225.62:43919)
0809 00:53:39.975577 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:39.975582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:39.975955 (+ 373us) server_negotiation.cc:407] Connection header received
0809 00:53:39.976175 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:39.976180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:39.976248 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:39.976363 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:39.977265 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:39.978355 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:39.979674 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:39.980148 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:39.980291 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:39.984946 (+ 4655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:34973: BlockingRecv error: recv got EOF from 127.9.225.1:34973 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 09 00:53:40 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:40 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:40.059389 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:40.048960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:50779)
0809 00:53:40.049640 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:40.049657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:40.049881 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:40.050262 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:40.050267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:40.050811 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:40.050994 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:40.051004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:40.052129 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:40.052133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:40.053248 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:40.053264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:40.054502 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:40.054511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:40.054673 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:40.055671 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:40.055692 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:40.059110 (+ 3418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":372,"threads_started":1}
W20260809 00:53:40.059660 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:40.049166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:50779 (local address 127.9.225.62:43919)
0809 00:53:40.049478 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:40.049486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:40.049800 (+ 314us) server_negotiation.cc:407] Connection header received
0809 00:53:40.050035 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:40.050041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:40.050118 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:40.050229 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:40.051159 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:40.051932 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:40.053473 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:40.054262 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:40.054428 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:40.059390 (+ 4962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:50779: BlockingRecv error: recv got EOF from 127.9.225.3:50779 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 09 00:53:40 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:40 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:40.538123 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:40.528636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:59097)
0809 00:53:40.529246 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:40.529270 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:40.529419 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:40.529801 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:40.529805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:40.530198 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:40.530368 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:40.530377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:40.531687 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:40.531692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:40.532661 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:40.532674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:40.533582 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:40.533587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:40.533740 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:40.534466 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:40.534484 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:40.537736 (+ 3252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":378,"threads_started":1}
W20260809 00:53:40.538357 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:40.528785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:59097 (local address 127.9.225.62:43919)
0809 00:53:40.528973 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:40.528977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:40.529364 (+ 387us) server_negotiation.cc:407] Connection header received
0809 00:53:40.529570 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:40.529574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:40.529663 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:40.529754 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:40.530544 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:40.531524 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:40.532854 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:40.533335 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:40.533530 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:40.538030 (+ 4500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:59097: BlockingRecv error: recv got EOF from 127.9.225.2:59097 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 00:53:40 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:40 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:40.996574 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:40.985972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:47751)
0809 00:53:40.986565 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:40.986582 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:40.986706 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:40.987041 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:40.987044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:40.987500 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:40.987698 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:40.987707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:40.988776 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:40.988780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:40.989788 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:40.989804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:40.990673 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:40.990679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:40.990815 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:40.991554 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:40.991576 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:40.996162 (+ 4586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":329,"threads_started":1}
W20260809 00:53:40.996735 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:40.986334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:47751 (local address 127.9.225.62:43919)
0809 00:53:40.986555 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:40.986559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:40.986661 (+ 102us) server_negotiation.cc:407] Connection header received
0809 00:53:40.986810 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:40.986814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:40.986882 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:40.986991 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:40.987857 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:40.988630 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:40.990034 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:40.990484 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:40.990599 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:40.996414 (+ 5815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:47751: BlockingRecv error: recv got EOF from 127.9.225.1:47751 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 09 00:53:41 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:41 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:41.070902 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:41.060641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:44003)
0809 00:53:41.061392 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:41.061417 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:41.061584 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:41.062007 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:41.062012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:41.062563 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:41.062725 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:41.062737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:41.063880 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:41.063885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:41.064853 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:41.064865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:41.065802 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:41.065808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:41.066025 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:41.066743 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:41.066772 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:41.070638 (+ 3866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":461,"threads_started":1}
W20260809 00:53:41.071440 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:41.060647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:44003 (local address 127.9.225.62:43919)
0809 00:53:41.060897 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:41.060903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:41.061530 (+ 627us) server_negotiation.cc:407] Connection header received
0809 00:53:41.061732 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:41.061737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:41.061846 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:41.061961 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:41.062914 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:41.063709 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:41.065020 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:41.065603 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:41.065763 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:41.071178 (+ 5415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:44003: BlockingRecv error: recv got EOF from 127.9.225.3:44003 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 00:53:41 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:41 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:41.548616 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:41.539306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:56015)
0809 00:53:41.539890 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:41.539912 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:41.540033 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:41.540425 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:41.540429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:41.540845 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:41.541033 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:41.541041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:41.542158 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:41.542162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:41.543179 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:41.543192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:41.543914 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:41.543919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:41.544052 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:41.544710 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:41.544734 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:41.548196 (+ 3462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":312,"threads_started":1}
W20260809 00:53:41.548671 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:41.539418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:56015 (local address 127.9.225.62:43919)
0809 00:53:41.539604 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:41.539608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:41.539989 (+ 381us) server_negotiation.cc:407] Connection header received
0809 00:53:41.540186 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:41.540190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:41.540279 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:41.540384 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:41.541162 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:41.542037 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:41.543307 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:41.543738 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:41.543858 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:41.548424 (+ 4566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:56015: BlockingRecv error: recv got EOF from 127.9.225.2:56015 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 00:53:41 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:42 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:42.007258 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:41.997835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:33315)
0809 00:53:41.998471 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:41.998488 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:41.998665 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:41.999048 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:41.999053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:41.999538 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:41.999706 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:41.999715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:42.000850 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:42.000854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:42.001891 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:42.001906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:42.002754 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:42.002760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:42.002975 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:42.003636 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:42.003655 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:42.006990 (+ 3335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":351,"threads_started":1}
W20260809 00:53:42.007761 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:41.997972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:33315 (local address 127.9.225.62:43919)
0809 00:53:41.998242 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:41.998251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:41.998584 (+ 333us) server_negotiation.cc:407] Connection header received
0809 00:53:41.998797 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:41.998803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:41.998883 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:41.998996 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:41.999868 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:42.000653 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:42.002055 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:42.002584 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:42.002732 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:42.007475 (+ 4743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:33315: BlockingRecv error: recv got EOF from 127.9.225.1:33315 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 09 00:53:42 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:42 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:42.081429 25136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:42.072184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:52725)
0809 00:53:42.072761 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:42.072783 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:42.072936 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:42.073384 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:42.073389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:42.074024 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:42.074224 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:42.074235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:42.075292 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:42.075297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:42.076232 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:42.076248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:42.077014 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:42.077019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:42.077147 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:42.077843 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:42.077862 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:42.081081 (+ 3219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":317,"threads_started":1}
W20260809 00:53:42.081729 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:42.072280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:52725 (local address 127.9.225.62:43919)
0809 00:53:42.072503 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:42.072508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:42.072882 (+ 374us) server_negotiation.cc:407] Connection header received
0809 00:53:42.073089 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:42.073094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:42.073188 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:42.073292 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:42.074375 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:42.075145 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:42.076386 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:42.076857 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:42.076961 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:42.081462 (+ 4501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:52725: BlockingRecv error: recv got EOF from 127.9.225.3:52725 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 09 00:53:42 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:42 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:42.560643 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:42.549936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:48387)
0809 00:53:42.550606 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:42.550631 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:42.550813 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:42.551298 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:42.551304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:42.551754 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:42.551922 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:42.551932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:42.553078 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:42.553084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:42.554086 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:42.554100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:42.555064 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:42.555070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:42.555276 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:42.556800 (+ 1524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:42.556834 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:42.560271 (+ 3437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":368,"threads_started":1}
W20260809 00:53:42.560855 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:42.550082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:48387 (local address 127.9.225.62:43919)
0809 00:53:42.550324 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:42.550331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:42.550745 (+ 414us) server_negotiation.cc:407] Connection header received
0809 00:53:42.551020 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:42.551026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:42.551136 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:42.551244 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:42.552095 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:42.552899 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:42.554262 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:42.554882 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:42.555041 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:42.560556 (+ 5515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:48387: BlockingRecv error: recv got EOF from 127.9.225.2:48387 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 00:53:43 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:43 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:43.018776 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:43.008483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:43115)
0809 00:53:43.009088 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:43.009105 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:43.009235 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:43.009663 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:43.009667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:43.010125 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:43.010320 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:43.010329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:43.011453 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:43.011459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:43.012482 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:43.012497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:43.013576 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:43.013583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:43.013720 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:43.014386 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:43.014405 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:43.018403 (+ 3998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":317,"threads_started":1}
W20260809 00:53:43.019229 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:43.008544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:43115 (local address 127.9.225.62:43919)
0809 00:53:43.008749 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:43.008754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:43.009212 (+ 458us) server_negotiation.cc:407] Connection header received
0809 00:53:43.009423 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:43.009428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:43.009500 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:43.009592 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:43.010476 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:43.011271 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:43.012643 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:43.013378 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:43.013505 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:43.018732 (+ 5227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:43115: BlockingRecv error: recv got EOF from 127.9.225.1:43115 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 09 00:53:43 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:43 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:43.092311 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:43.082709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:37763)
0809 00:53:43.083516 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:43.083540 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:43.083703 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:43.084148 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:43.084154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:43.084702 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:43.084878 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:43.084888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:43.086322 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:43.086327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:43.087283 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:43.087298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:43.087984 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:43.087988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:43.088132 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:43.088791 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:43.088812 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:43.092044 (+ 3232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":513,"threads_started":1}
W20260809 00:53:43.092672 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:43.082827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:37763 (local address 127.9.225.62:43919)
0809 00:53:43.083057 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:43.083062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:43.083647 (+ 585us) server_negotiation.cc:407] Connection header received
0809 00:53:43.083853 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:43.083859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:43.083967 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:43.084060 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:43.085037 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:43.086113 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:43.087428 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:43.087836 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:43.087943 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:43.092336 (+ 4393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:37763: BlockingRecv error: recv got EOF from 127.9.225.3:37763 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 09 00:53:43 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:43 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:43.572173 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:43.561963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:48123)
0809 00:53:43.562612 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:43.562634 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:43.562769 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:43.563231 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:43.563235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:43.563714 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:43.563899 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:43.563909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:43.565199 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:43.565207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:43.566695 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:43.566713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:43.567495 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:43.567499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:43.567630 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:43.568338 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:43.568358 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:43.571910 (+ 3552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":344,"threads_started":1}
W20260809 00:53:43.572576 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:43.562166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:48123 (local address 127.9.225.62:43919)
0809 00:53:43.562355 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:43.562360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:43.562721 (+ 361us) server_negotiation.cc:407] Connection header received
0809 00:53:43.562888 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:43.562892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:43.563033 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:43.563147 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:43.564126 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:43.565012 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:43.566832 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:43.567298 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:43.567410 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:43.572248 (+ 4838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:48123: BlockingRecv error: recv got EOF from 127.9.225.2:48123 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 09 00:53:44 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:44 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:44.032943 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:44.020099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:42117)
0809 00:53:44.021195 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:44.021221 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:44.021439 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:44.021914 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:44.021922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:44.022602 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:44.022934 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:44.022952 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:44.024354 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:44.024363 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:44.026453 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:44.026481 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:44.027477 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:44.027485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:44.027690 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:44.028437 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:44.028457 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:44.032456 (+ 3999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":370,"threads_started":1}
W20260809 00:53:44.032990 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:44.020229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:42117 (local address 127.9.225.62:43919)
0809 00:53:44.020450 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:44.020456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:44.021314 (+ 858us) server_negotiation.cc:407] Connection header received
0809 00:53:44.021626 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:44.021631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:44.021723 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:44.021863 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:44.023204 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:44.024165 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:44.026654 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:44.027293 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:44.027465 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:44.032768 (+ 5303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:42117: BlockingRecv error: recv got EOF from 127.9.225.1:42117 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 09 00:53:44 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:44 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:44.103964 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:44.093553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:42509)
0809 00:53:44.094223 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:44.094247 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:44.094411 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:44.094811 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:44.094815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:44.095238 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:44.095418 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:44.095428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:44.096629 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:44.096636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:44.097868 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:44.097885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:44.098815 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:44.098821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:44.098991 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:44.099718 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:44.099737 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:44.103678 (+ 3941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":330,"threads_started":1}
W20260809 00:53:44.104261 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:44.094158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:42509 (local address 127.9.225.62:43919)
0809 00:53:44.094490 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:44.094495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:44.094513 (+ 18us) server_negotiation.cc:407] Connection header received
0809 00:53:44.094571 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:44.094575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:44.094639 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:44.094747 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:44.095596 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:44.096393 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:44.098058 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:44.098555 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:44.098732 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:44.103982 (+ 5250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:42509: BlockingRecv error: recv got EOF from 127.9.225.3:42509 (error 108)
Metrics: {"server-negotiator.queue_time_us":200}
Aug 09 00:53:44 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:44 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:44.588486 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:44.573655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:59613)
0809 00:53:44.574283 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:44.574315 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:44.574541 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:44.574974 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:44.574978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:44.575446 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:44.575644 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:44.575653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:44.576750 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:44.576756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:44.577747 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:44.577761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:44.578619 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:44.578625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:44.578824 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:44.584853 (+ 6029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:44.584877 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:44.588101 (+ 3224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":357,"threads_started":1}
W20260809 00:53:44.588662 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:44.573718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:59613 (local address 127.9.225.62:43919)
0809 00:53:44.574056 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:44.574065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:44.574426 (+ 361us) server_negotiation.cc:407] Connection header received
0809 00:53:44.574697 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:44.574703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:44.574805 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:44.574924 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:44.575810 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:44.576578 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:44.577936 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:44.578440 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:44.578596 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:44.588378 (+ 9782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:59613: BlockingRecv error: recv got EOF from 127.9.225.2:59613 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 09 00:53:45 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:45 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.1@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:45.044279 25150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:45.034623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.1:44939)
0809 00:53:45.035537 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:45.035554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:45.035674 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:45.036062 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:45.036067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:45.036556 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:45.036875 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:45.036889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.037968 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.037973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:45.039213 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:45.039229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.040026 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.040033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:45.040226 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:45.040888 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:45.040907 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:45.044002 (+ 3095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":664,"threads_started":1}
W20260809 00:53:45.044569 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:45.034700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:44939 (local address 127.9.225.62:43919)
0809 00:53:45.034932 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:45.034939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:45.035638 (+ 699us) server_negotiation.cc:407] Connection header received
0809 00:53:45.035772 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:45.035776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:45.035863 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:45.035968 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:45.037028 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.037823 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.039372 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.039818 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.039960 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:45.044269 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.1:44939: BlockingRecv error: recv got EOF from 127.9.225.1:44939 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 00:53:45 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:45 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.3@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:45.114881 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:45.105189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.3:36699)
0809 00:53:45.105782 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:45.105798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:45.105967 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:45.106348 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:45.106353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:45.106818 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:45.107022 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:45.107034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.108146 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.108152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:45.109126 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:45.109140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.110115 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.110121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:45.110310 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:45.111272 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:45.111299 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:45.114590 (+ 3291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":340,"threads_started":1}
W20260809 00:53:45.115312 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:45.105390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:36699 (local address 127.9.225.62:43919)
0809 00:53:45.105639 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:45.105647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:45.105895 (+ 248us) server_negotiation.cc:407] Connection header received
0809 00:53:45.106120 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:45.106129 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:45.106197 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:45.106309 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:45.107192 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.107976 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.109288 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.109946 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.110095 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:45.114981 (+ 4886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.3:36699: BlockingRecv error: recv got EOF from 127.9.225.3:36699 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 09 00:53:45 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 00:53:45 dist-test-slave-j2vl krb5kdc[23494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.225.2@KRBTEST.COM for oryx/127.9.225.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 00:53:45.599874 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:45.589969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43919 (local address 127.9.225.2:41395)
0809 00:53:45.590589 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:45.590612 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:45.590748 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:45.591129 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:45.591132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:45.591579 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:45.591801 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:45.591811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.593046 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.593051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:45.594067 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:45.594081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.594876 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.594881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:45.595023 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:45.595680 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:45.595703 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:45.599448 (+ 3745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.225.62:43919: Server oryx/127.9.225.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":342,"threads_started":1}
W20260809 00:53:45.600070 25079 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:45.590129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:41395 (local address 127.9.225.62:43919)
0809 00:53:45.590320 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:45.590326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:45.590725 (+ 399us) server_negotiation.cc:407] Connection header received
0809 00:53:45.590862 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:45.590873 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:45.590978 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:45.591074 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:45.591987 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.592880 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.594235 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.594724 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.594834 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:45.599759 (+ 4925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.225.2:41395: BlockingRecv error: recv got EOF from 127.9.225.2:41395 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 00:53:45.930227 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 24436
I20260809 00:53:45.945611 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:45.931761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.9.225.1:46599)
0809 00:53:45.932484 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:45.932493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:45.932523 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:53:45.932593 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:45.932600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:45.932856 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:45.933023 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:45.934888 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.936252 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.937657 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:45.938346 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:45.938467 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:45.941557 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:45.941607 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:45.941631 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:45.941700 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:45.944807 (+ 3107us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.225.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 00:53:45.945271 (+ 464us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38434: Request ticket server kudu/127.9.225.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":398,"threads_started":1}
W20260809 00:53:45.945923 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 24436: Not authorized: Client connection negotiation failed: client connection to 127.9.225.1:46599: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.225.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 00:53:45.945953 25155 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38434: Request ticket server kudu/127.9.225.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 00:53:45.946163 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 24436
I20260809 00:53:45.997329 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 24673
I20260809 00:53:46.011507 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:45.998668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.9.225.2:36737)
0809 00:53:45.999263 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:45.999272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:45.999304 (+ 32us) server_negotiation.cc:407] Connection header received
0809 00:53:45.999369 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:45.999376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:45.999624 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:45.999784 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:46.001285 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:46.002759 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:46.004223 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:46.004856 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:46.004985 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:46.007896 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:46.007907 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:46.007928 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:46.007986 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:46.010896 (+ 2910us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.225.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 00:53:46.011229 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34480: Request ticket server kudu/127.9.225.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":290,"threads_started":1}
W20260809 00:53:46.011656 25156 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34480: Request ticket server kudu/127.9.225.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 00:53:46.011860 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 24673: Not authorized: Client connection negotiation failed: client connection to 127.9.225.2:36737: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.225.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 00:53:46.012017 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 24673
I20260809 00:53:46.047242 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 24883
I20260809 00:53:46.062968 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:46.048366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.9.225.3:40821)
0809 00:53:46.049002 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:46.049010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:46.049041 (+ 31us) server_negotiation.cc:407] Connection header received
0809 00:53:46.049111 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:46.049118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:46.049468 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:46.049644 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:46.051175 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:46.052272 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:46.053732 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:46.054338 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:46.054463 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:46.058346 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:46.058361 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:46.058388 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:46.058452 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:46.062157 (+ 3705us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.225.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 00:53:46.062567 (+ 410us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57316: Request ticket server kudu/127.9.225.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":327,"threads_started":1}
W20260809 00:53:46.063198 25158 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57316: Request ticket server kudu/127.9.225.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 00:53:46.063570 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 24883: Not authorized: Client connection negotiation failed: client connection to 127.9.225.3:40821: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.225.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 00:53:46.063771 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 24883
I20260809 00:53:46.091409 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 24329
I20260809 00:53:46.093009 24425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:46.221328 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 24329
2026-08-09T00:53:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22517 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T00:53:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:53:46Z Disabled control of system clock
I20260809 00:53:46.333998 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43827
--webserver_interface=127.9.225.62
--webserver_port=0
--builtin_ntp_servers=127.9.225.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:43827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 00:53:46.736763 25164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:46.737135 25164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:46.737221 25164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:46.746240 25164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:53:46.746402 25164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:46.746518 25164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:46.746623 25164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:53:46.746680 25164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:53:46.757422 25164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:43827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:46.758965 25164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:46.761703 25164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:46.775875 25173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:46.775902 25177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:46.775971 25182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:46.776866 25164 server_base.cc:1034] running on GCE node
I20260809 00:53:46.778023 25164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:46.779704 25164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:46.780953 25164 hybrid_clock.cc:648] HybridClock initialized: now 1786236826780904 us; error 33 us; skew 500 ppm
I20260809 00:53:46.781418 25164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:53:46.784286 25164 webserver.cc:460] Webserver started at http://127.9.225.62:40063/ using document root <none> and password file <none>
I20260809 00:53:46.785141 25164 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:46.785247 25164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:46.785648 25164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:46.789163 25164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "97b02a9edae74f6fbde8c5c7b096f5e3"
format_stamp: "Formatted at 2026-08-09 00:53:46 on dist-test-slave-j2vl"
server_key: "e93a8bf186d6c3850004212faf7291a6"
server_key_iv: "45e731f7cc3776fcdfb5ae4197610489"
server_key_version: "encryptionkey@0"
I20260809 00:53:46.790181 25164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "97b02a9edae74f6fbde8c5c7b096f5e3"
format_stamp: "Formatted at 2026-08-09 00:53:46 on dist-test-slave-j2vl"
server_key: "e93a8bf186d6c3850004212faf7291a6"
server_key_iv: "45e731f7cc3776fcdfb5ae4197610489"
server_key_version: "encryptionkey@0"
I20260809 00:53:46.796454 25164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 00:53:46.800896 25188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:46.802665 25164 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:53:46.802855 25164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "97b02a9edae74f6fbde8c5c7b096f5e3"
format_stamp: "Formatted at 2026-08-09 00:53:46 on dist-test-slave-j2vl"
server_key: "e93a8bf186d6c3850004212faf7291a6"
server_key_iv: "45e731f7cc3776fcdfb5ae4197610489"
server_key_version: "encryptionkey@0"
I20260809 00:53:46.803081 25164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:46.820459 25164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:46.821525 25164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:46.821801 25164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:46.843561 25164 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:43827
I20260809 00:53:46.843581 25271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:43827 every 8 connection(s)
I20260809 00:53:46.845254 25164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:53:46.849236 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 25164
I20260809 00:53:46.849500 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:53:46.850404 25273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:46.850428 10116 external_mini_cluster.cc:1428] Setting key c310a1dbacfce9af2a2e0b058558bb8c
I20260809 00:53:46.865314 25273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3: Bootstrap starting.
I20260809 00:53:46.866297 25278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:46.853523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.9.225.62:43827)
0809 00:53:46.854634 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:46.854647 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:46.854695 (+ 48us) server_negotiation.cc:407] Connection header received
0809 00:53:46.856355 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:46.856381 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:46.856946 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:46.857434 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:46.858127 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:46.859470 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:46.860873 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:46.861718 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:46.861970 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:46.863488 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:46.863498 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:46.863517 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:46.864145 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:46.864235 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:46.864262 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:46.864419 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:46.864670 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 00:53:46.865063 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":376,"threads_started":1}
I20260809 00:53:46.869122 25273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:46.870380 25273 log.cc:824] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:46.874008 25273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3: No bootstrap required, opened a new log
I20260809 00:53:46.882544 25273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b02a9edae74f6fbde8c5c7b096f5e3" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43827 } }
I20260809 00:53:46.883039 25273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:46.883210 25273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b02a9edae74f6fbde8c5c7b096f5e3, State: Initialized, Role: FOLLOWER
I20260809 00:53:46.883955 25273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b02a9edae74f6fbde8c5c7b096f5e3" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43827 } }
I20260809 00:53:46.884233 25273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:53:46.884413 25273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:53:46.884613 25273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:46.887121 25273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b02a9edae74f6fbde8c5c7b096f5e3" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43827 } }
I20260809 00:53:46.887797 25273 leader_election.cc:302] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97b02a9edae74f6fbde8c5c7b096f5e3; no voters:
I20260809 00:53:46.888477 25273 leader_election.cc:288] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:53:46.888662 25282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:46.889485 25282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [term 1 LEADER]: Becoming Leader. State: Replica: 97b02a9edae74f6fbde8c5c7b096f5e3, State: Running, Role: LEADER
I20260809 00:53:46.890177 25282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b02a9edae74f6fbde8c5c7b096f5e3" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43827 } }
I20260809 00:53:46.890905 25273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:53:46.893473 25283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97b02a9edae74f6fbde8c5c7b096f5e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b02a9edae74f6fbde8c5c7b096f5e3" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43827 } } }
I20260809 00:53:46.893709 25284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97b02a9edae74f6fbde8c5c7b096f5e3. Latest consensus state: current_term: 1 leader_uuid: "97b02a9edae74f6fbde8c5c7b096f5e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b02a9edae74f6fbde8c5c7b096f5e3" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43827 } } }
I20260809 00:53:46.893996 25284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:46.894001 25283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3 [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:46.901435 25298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:53:46.908099 25298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:53:46.919495 25298 catalog_manager.cc:1347] Generated new cluster ID: 388146303bb54db0b35c7c72afbed189
I20260809 00:53:46.919664 25298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:53:46.978341 25298 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:53:46.980142 25298 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:53:46.988404 25298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97b02a9edae74f6fbde8c5c7b096f5e3: Generated new TSK 0
I20260809 00:53:46.989581 25298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:53:46.998230 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43827
--builtin_ntp_servers=127.9.225.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 00:53:47.396366 25313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:47.396770 25313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:47.396914 25313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:47.404891 25313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:47.405035 25313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:47.405223 25313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:53:47.415452 25313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:47.417073 25313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:47.419226 25313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:47.434167 25318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:47.434273 25319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:47.434202 25322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:47.435711 25313 server_base.cc:1034] running on GCE node
I20260809 00:53:47.436697 25313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:47.438002 25313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:47.439265 25313 hybrid_clock.cc:648] HybridClock initialized: now 1786236827439192 us; error 72 us; skew 500 ppm
I20260809 00:53:47.439656 25313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:53:47.442502 25313 webserver.cc:460] Webserver started at http://127.9.225.1:43491/ using document root <none> and password file <none>
I20260809 00:53:47.443449 25313 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:47.443616 25313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:47.444000 25313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:47.447399 25313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "680be1a1961a4d56a246da0627bde3f5"
format_stamp: "Formatted at 2026-08-09 00:53:47 on dist-test-slave-j2vl"
server_key: "472bab87f0f027b1b64d904975421cac"
server_key_iv: "527919de3b9d987a75b50a69e5228128"
server_key_version: "encryptionkey@0"
I20260809 00:53:47.448360 25313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "680be1a1961a4d56a246da0627bde3f5"
format_stamp: "Formatted at 2026-08-09 00:53:47 on dist-test-slave-j2vl"
server_key: "472bab87f0f027b1b64d904975421cac"
server_key_iv: "527919de3b9d987a75b50a69e5228128"
server_key_version: "encryptionkey@0"
I20260809 00:53:47.454586 25313 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 00:53:47.458766 25332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:47.460383 25313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 00:53:47.460624 25313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "680be1a1961a4d56a246da0627bde3f5"
format_stamp: "Formatted at 2026-08-09 00:53:47 on dist-test-slave-j2vl"
server_key: "472bab87f0f027b1b64d904975421cac"
server_key_iv: "527919de3b9d987a75b50a69e5228128"
server_key_version: "encryptionkey@0"
I20260809 00:53:47.460891 25313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:47.473097 25313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:47.474195 25313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:47.474529 25313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:47.475664 25313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:47.477392 25313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:47.477541 25313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 00:53:47.477694 25313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:47.477792 25313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:47.515131 25313 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:34041
I20260809 00:53:47.515194 25504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:34041 every 8 connection(s)
I20260809 00:53:47.516762 25313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:53:47.518767 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 25313
I20260809 00:53:47.518989 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:53:47.519711 10116 external_mini_cluster.cc:1428] Setting key 6d0181addada0d9b9c67ba635f683686
I20260809 00:53:47.522986 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43827
--builtin_ntp_servers=127.9.225.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 00:53:47.533030 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:47.520057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:44089 (local address 127.9.225.62:43827)
0809 00:53:47.520675 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:47.520684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:47.522337 (+ 1653us) server_negotiation.cc:407] Connection header received
0809 00:53:47.524343 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:47.524353 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:47.524472 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:47.524600 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:47.525936 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:47.527118 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:47.528962 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:47.529745 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:47.529912 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:47.531880 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:47.531889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:47.531897 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:47.531926 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:47.531984 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:47.532004 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:47.532108 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:47.532430 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 00:53:47.532573 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":859,"threads_started":1}
I20260809 00:53:47.534133 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:47.520770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43827 (local address 127.9.225.1:44089)
0809 00:53:47.522135 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:47.522204 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:47.524016 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:47.524844 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:47.524854 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:47.524994 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:47.525731 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:47.525751 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:47.527337 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:47.527341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:47.528762 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:47.528778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:47.530155 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:47.530162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:47.530413 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:47.531439 (+ 1026us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:53:47.531442 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:47.531495 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:53:47.531607 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:47.532146 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:47.532156 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:47.532165 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 00:53:47.532369 (+ 204us) client_negotiation.cc:240] Negotiation successful
0809 00:53:47.532799 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":474,"threads_started":1}
I20260809 00:53:47.536302 25505 heartbeater.cc:347] Connected to a master server at 127.9.225.62:43827
I20260809 00:53:47.536693 25505 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:47.537678 25505 heartbeater.cc:511] Master 127.9.225.62:43827 requested a full tablet report, sending...
I20260809 00:53:47.540140 25219 ts_manager.cc:194] Registered new tserver with Master: 680be1a1961a4d56a246da0627bde3f5 (127.9.225.1:34041)
I20260809 00:53:47.541833 25219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.225.1:44089
W20260809 00:53:47.904542 25510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:47.904953 25510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:47.905118 25510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:47.913142 25510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:47.913347 25510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:47.913569 25510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:53:47.923486 25510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:47.925066 25510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:47.927101 25510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:47.941601 25523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:47.941641 25519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:47.942301 25517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:47.942802 25510 server_base.cc:1034] running on GCE node
I20260809 00:53:47.943503 25510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:47.944732 25510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:47.945963 25510 hybrid_clock.cc:648] HybridClock initialized: now 1786236827945898 us; error 43 us; skew 500 ppm
I20260809 00:53:47.946297 25510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:53:47.948884 25510 webserver.cc:460] Webserver started at http://127.9.225.2:44909/ using document root <none> and password file <none>
I20260809 00:53:47.949770 25510 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:47.949874 25510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:47.950198 25510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:47.953317 25510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "1594a243605b40d295e0bcbd517748b3"
format_stamp: "Formatted at 2026-08-09 00:53:47 on dist-test-slave-j2vl"
server_key: "81d9be1de26f796a8b6c021bca8a2198"
server_key_iv: "669c44c536e756ebdcc1fee57b7e810c"
server_key_version: "encryptionkey@0"
I20260809 00:53:47.954257 25510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "1594a243605b40d295e0bcbd517748b3"
format_stamp: "Formatted at 2026-08-09 00:53:47 on dist-test-slave-j2vl"
server_key: "81d9be1de26f796a8b6c021bca8a2198"
server_key_iv: "669c44c536e756ebdcc1fee57b7e810c"
server_key_version: "encryptionkey@0"
I20260809 00:53:47.960352 25510 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 00:53:47.964403 25535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:47.966220 25510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 00:53:47.966418 25510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "1594a243605b40d295e0bcbd517748b3"
format_stamp: "Formatted at 2026-08-09 00:53:47 on dist-test-slave-j2vl"
server_key: "81d9be1de26f796a8b6c021bca8a2198"
server_key_iv: "669c44c536e756ebdcc1fee57b7e810c"
server_key_version: "encryptionkey@0"
I20260809 00:53:47.966691 25510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:47.987841 25510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:47.988984 25510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:47.989326 25510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:47.990586 25510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:47.992520 25510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:47.992650 25510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:47.992806 25510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:47.992903 25510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:48.030036 25510 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:40221
I20260809 00:53:48.030138 25728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:40221 every 8 connection(s)
I20260809 00:53:48.031558 25510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:53:48.039094 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 25510
I20260809 00:53:48.039286 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:53:48.039775 10116 external_mini_cluster.cc:1428] Setting key abf39437c8455340a1462831e0a00bb2
I20260809 00:53:48.043107 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43827
--builtin_ntp_servers=127.9.225.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 00:53:48.047055 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:48.034480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:48371 (local address 127.9.225.62:43827)
0809 00:53:48.034995 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:48.035000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:48.036415 (+ 1415us) server_negotiation.cc:407] Connection header received
0809 00:53:48.038407 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:48.038413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:48.038524 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:48.038680 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:48.040017 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.040985 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.043360 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.044120 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.044282 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:48.045986 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:48.045995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:48.046003 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:48.046032 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:48.046089 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:48.046109 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:48.046211 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:48.046457 (+ 246us) server_negotiation.cc:299] Negotiation successful
0809 00:53:48.046591 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":289,"threads_started":1}
I20260809 00:53:48.048048 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:48.035215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43827 (local address 127.9.225.2:48371)
0809 00:53:48.036235 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:48.036274 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:48.038103 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:48.038934 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:48.038943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:48.039046 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:48.039788 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:48.039814 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.041153 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.041157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:48.043143 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:48.043162 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.044372 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.044378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:48.044669 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:48.045588 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:53:48.045591 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:48.045642 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:53:48.045749 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:48.046216 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:48.046224 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:48.046231 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 00:53:48.046389 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 00:53:48.046759 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":310,"threads_started":1}
I20260809 00:53:48.049808 25729 heartbeater.cc:347] Connected to a master server at 127.9.225.62:43827
I20260809 00:53:48.050194 25729 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:48.051293 25729 heartbeater.cc:511] Master 127.9.225.62:43827 requested a full tablet report, sending...
I20260809 00:53:48.053226 25219 ts_manager.cc:194] Registered new tserver with Master: 1594a243605b40d295e0bcbd517748b3 (127.9.225.2:40221)
I20260809 00:53:48.054126 25219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.225.2:48371
W20260809 00:53:48.449306 25733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:48.449749 25733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:48.449915 25733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:48.457906 25733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:48.458030 25733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:48.458206 25733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:53:48.471094 25733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:48.472657 25733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:48.474887 25733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:48.490242 25745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:48.490316 25746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:48.490340 25748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:48.495283 25733 server_base.cc:1034] running on GCE node
I20260809 00:53:48.496156 25733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:48.497586 25733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:48.498862 25733 hybrid_clock.cc:648] HybridClock initialized: now 1786236828498782 us; error 85 us; skew 500 ppm
I20260809 00:53:48.499285 25733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:53:48.502067 25733 webserver.cc:460] Webserver started at http://127.9.225.3:44635/ using document root <none> and password file <none>
I20260809 00:53:48.502952 25733 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:48.503114 25733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:48.503518 25733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:48.506886 25733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "d8920222d3f5405fb2a0192cb8a4c736"
format_stamp: "Formatted at 2026-08-09 00:53:48 on dist-test-slave-j2vl"
server_key: "ed884ad3c896c917b849a90d4b73e607"
server_key_iv: "be9852382586af51f84a27e686aaec9e"
server_key_version: "encryptionkey@0"
I20260809 00:53:48.507911 25733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "d8920222d3f5405fb2a0192cb8a4c736"
format_stamp: "Formatted at 2026-08-09 00:53:48 on dist-test-slave-j2vl"
server_key: "ed884ad3c896c917b849a90d4b73e607"
server_key_iv: "be9852382586af51f84a27e686aaec9e"
server_key_version: "encryptionkey@0"
I20260809 00:53:48.514995 25733 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260809 00:53:48.519466 25757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:48.521162 25733 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260809 00:53:48.521399 25733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "d8920222d3f5405fb2a0192cb8a4c736"
format_stamp: "Formatted at 2026-08-09 00:53:48 on dist-test-slave-j2vl"
server_key: "ed884ad3c896c917b849a90d4b73e607"
server_key_iv: "be9852382586af51f84a27e686aaec9e"
server_key_version: "encryptionkey@0"
I20260809 00:53:48.521667 25733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:48.545805 25505 heartbeater.cc:503] Master 127.9.225.62:43827 was elected leader, sending a full tablet report...
I20260809 00:53:48.567029 25733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:48.568150 25733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:48.568477 25733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:48.569730 25733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:48.571588 25733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:48.571748 25733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:48.571895 25733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:48.571988 25733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:48.609232 25733 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:34565
I20260809 00:53:48.609302 25930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:34565 every 8 connection(s)
I20260809 00:53:48.610700 25733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:53:48.620311 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 25733
I20260809 00:53:48.620460 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:53:48.620972 10116 external_mini_cluster.cc:1428] Setting key c7a260f9e2bce33d926383276159cc2d
I20260809 00:53:48.624722 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:48.613556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:41009 (local address 127.9.225.62:43827)
0809 00:53:48.614004 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:48.614010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:48.615388 (+ 1378us) server_negotiation.cc:407] Connection header received
0809 00:53:48.617310 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:48.617316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:48.617449 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:48.617581 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:48.618798 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.619590 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.621538 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.622091 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.622239 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:48.623780 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:48.623785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:48.623788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:48.623808 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:48.623852 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:48.623865 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:48.623972 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:48.624228 (+ 256us) server_negotiation.cc:299] Negotiation successful
0809 00:53:48.624341 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260809 00:53:48.626037 25935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:48.614207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43827 (local address 127.9.225.3:41009)
0809 00:53:48.615201 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:48.615247 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:48.616986 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:48.617849 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:48.617859 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:48.617975 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:48.618606 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:48.618623 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.619764 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.619769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:48.621320 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:48.621334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.622352 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.622359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:48.622621 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:48.623408 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:53:48.623410 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:48.623459 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:53:48.623559 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:48.623973 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:48.623987 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:48.623998 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 00:53:48.624219 (+ 221us) client_negotiation.cc:240] Negotiation successful
0809 00:53:48.624703 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":292,"threads_started":1}
I20260809 00:53:48.627527 25931 heartbeater.cc:347] Connected to a master server at 127.9.225.62:43827
I20260809 00:53:48.627879 25931 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:48.628762 25931 heartbeater.cc:511] Master 127.9.225.62:43827 requested a full tablet report, sending...
I20260809 00:53:48.630395 25219 ts_manager.cc:194] Registered new tserver with Master: d8920222d3f5405fb2a0192cb8a4c736 (127.9.225.3:34565)
I20260809 00:53:48.631287 25219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.225.3:41009
I20260809 00:53:48.637137 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 00:53:48.644107 25933 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:48.642397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.9.225.62:43827)
0809 00:53:48.642617 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:48.642622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:48.643111 (+ 489us) server_negotiation.cc:407] Connection header received
0809 00:53:48.643259 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:48.643263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:48.643334 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:48.643436 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:48.643838 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49124: BlockingRecv error: recv got EOF from 127.0.0.1:49124 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 00:53:48.644639 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 25313
I20260809 00:53:48.654210 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:48.645865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.9.225.1:34041)
0809 00:53:48.646539 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:48.646547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:48.646578 (+ 31us) server_negotiation.cc:407] Connection header received
0809 00:53:48.646643 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:48.646650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:48.646958 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:48.647100 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:48.647575 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.648683 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.650280 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.650991 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.651105 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:48.652642 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:48.652651 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:48.652669 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:48.653334 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:48.653464 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:48.653486 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:48.653585 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:48.653734 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 00:53:48.653873 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":339,"threads_started":1}
I20260809 00:53:48.655836 25498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:48.743546 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 25313
I20260809 00:53:48.781890 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 25510
I20260809 00:53:48.790984 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:48.783011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.9.225.2:40221)
0809 00:53:48.783675 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:48.783683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:48.783713 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:53:48.783778 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:48.783785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:48.784066 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:48.784200 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:48.784611 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.785748 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.787088 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.787707 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.787816 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:48.789098 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:48.789108 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:48.789127 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:48.789899 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:48.789982 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:48.790006 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:48.790104 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:48.790332 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 00:53:48.790522 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":376,"threads_started":1}
I20260809 00:53:48.792603 25723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:48.880446 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 25510
I20260809 00:53:48.915984 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 25733
I20260809 00:53:48.925489 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:48.917115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.9.225.3:34565)
0809 00:53:48.917883 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:48.917891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:48.917927 (+ 36us) server_negotiation.cc:407] Connection header received
0809 00:53:48.917999 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:48.918007 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:48.918268 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:48.918463 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:48.919013 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.920156 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.921596 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:48.922280 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:48.922447 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:48.923756 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:48.923767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:48.923784 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:48.924451 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:48.924505 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:48.924528 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:48.924677 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:48.924849 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 00:53:48.925003 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":391,"threads_started":1}
I20260809 00:53:48.927172 25921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:49.014257 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 25733
I20260809 00:53:49.044420 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 25164
I20260809 00:53:49.045873 25266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:49.159713 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 25164
2026-08-09T00:53:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2940 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T00:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:53:49Z Disabled control of system clock
I20260809 00:53:49.284482 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:36775
--webserver_interface=127.9.225.62
--webserver_port=0
--builtin_ntp_servers=127.9.225.20:41521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:36775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 00:53:49.700125 25958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:49.700560 25958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:49.700729 25958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:49.708586 25958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:53:49.708779 25958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:49.708876 25958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:49.708992 25958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:53:49.709074 25958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:53:49.719352 25958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:36775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:36775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:49.720968 25958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:49.723083 25958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:49.736712 25972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:49.736757 25969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:49.736786 25968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:49.737654 25958 server_base.cc:1034] running on GCE node
I20260809 00:53:49.738680 25958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:49.740433 25958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:49.741693 25958 hybrid_clock.cc:648] HybridClock initialized: now 1786236829741641 us; error 51 us; skew 500 ppm
I20260809 00:53:49.742122 25958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:53:49.749264 25958 webserver.cc:460] Webserver started at http://127.9.225.62:44749/ using document root <none> and password file <none>
I20260809 00:53:49.750226 25958 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:49.750394 25958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:49.750785 25958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:49.754326 25958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "520de27dd6f2434abe0eeba93cca05e7"
format_stamp: "Formatted at 2026-08-09 00:53:49 on dist-test-slave-j2vl"
server_key: "e5d5869320d22efb1a4bcc48aafacd05"
server_key_iv: "14dff7860d3d3bd6754d95395373b938"
server_key_version: "encryptionkey@0"
I20260809 00:53:49.755667 25958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "520de27dd6f2434abe0eeba93cca05e7"
format_stamp: "Formatted at 2026-08-09 00:53:49 on dist-test-slave-j2vl"
server_key: "e5d5869320d22efb1a4bcc48aafacd05"
server_key_iv: "14dff7860d3d3bd6754d95395373b938"
server_key_version: "encryptionkey@0"
I20260809 00:53:49.762010 25958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260809 00:53:49.766229 25979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:49.767907 25958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 00:53:49.768162 25958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "520de27dd6f2434abe0eeba93cca05e7"
format_stamp: "Formatted at 2026-08-09 00:53:49 on dist-test-slave-j2vl"
server_key: "e5d5869320d22efb1a4bcc48aafacd05"
server_key_iv: "14dff7860d3d3bd6754d95395373b938"
server_key_version: "encryptionkey@0"
I20260809 00:53:49.768471 25958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:49.775178 25958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:49.776154 25958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:49.776518 25958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:49.797291 25958 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:36775
I20260809 00:53:49.797422 26070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:36775 every 8 connection(s)
I20260809 00:53:49.798962 25958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:53:49.803668 26071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:49.808585 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 25958
I20260809 00:53:49.808856 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:53:49.809798 10116 external_mini_cluster.cc:1428] Setting key cfffacb90af804d13061e66280d0e72f
I20260809 00:53:49.818070 26071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7: Bootstrap starting.
I20260809 00:53:49.820348 26082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:49.812320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.9.225.62:36775)
0809 00:53:49.813534 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:49.813547 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:49.813597 (+ 50us) server_negotiation.cc:407] Connection header received
0809 00:53:49.816159 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:49.816194 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:49.816747 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:49.817214 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:49.818483 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:49.818490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:49.818504 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:49.818519 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:49.818587 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:49.818604 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:49.818864 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:49.819014 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 00:53:49.819305 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":405,"threads_started":1}
I20260809 00:53:49.822306 26071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:49.823589 26071 log.cc:824] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:49.827204 26071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7: No bootstrap required, opened a new log
I20260809 00:53:49.835800 26071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520de27dd6f2434abe0eeba93cca05e7" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36775 } }
I20260809 00:53:49.836294 26071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:49.836373 26071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 520de27dd6f2434abe0eeba93cca05e7, State: Initialized, Role: FOLLOWER
I20260809 00:53:49.837020 26071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520de27dd6f2434abe0eeba93cca05e7" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36775 } }
I20260809 00:53:49.837262 26071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:53:49.837462 26071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:53:49.837631 26071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:49.840024 26071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520de27dd6f2434abe0eeba93cca05e7" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36775 } }
I20260809 00:53:49.840616 26071 leader_election.cc:302] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 520de27dd6f2434abe0eeba93cca05e7; no voters:
I20260809 00:53:49.841238 26071 leader_election.cc:288] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:53:49.841463 26085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:49.842893 26085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [term 1 LEADER]: Becoming Leader. State: Replica: 520de27dd6f2434abe0eeba93cca05e7, State: Running, Role: LEADER
I20260809 00:53:49.843585 26085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520de27dd6f2434abe0eeba93cca05e7" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36775 } }
I20260809 00:53:49.843966 26071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:53:49.847070 26089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 520de27dd6f2434abe0eeba93cca05e7. Latest consensus state: current_term: 1 leader_uuid: "520de27dd6f2434abe0eeba93cca05e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520de27dd6f2434abe0eeba93cca05e7" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36775 } } }
I20260809 00:53:49.847141 26088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "520de27dd6f2434abe0eeba93cca05e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520de27dd6f2434abe0eeba93cca05e7" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36775 } } }
I20260809 00:53:49.847564 26089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:49.847611 26088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7 [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:49.851382 26100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:53:49.857425 26100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:53:49.868603 26100 catalog_manager.cc:1347] Generated new cluster ID: 17ca585f1f494aa7934a06303a044d1c
I20260809 00:53:49.868841 26100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:53:49.890002 26100 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:53:49.890599 26100 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:53:49.906519 26100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 520de27dd6f2434abe0eeba93cca05e7: Generated new TSK 0
I20260809 00:53:49.908071 26100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:53:49.915484 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:36775
--builtin_ntp_servers=127.9.225.20:41521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 00:53:50.329725 26121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:50.330060 26121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:50.330160 26121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:50.337997 26121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:50.338128 26121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:50.338245 26121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:53:50.348325 26121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:36775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:50.350065 26121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:50.352202 26121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:50.368223 26130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:50.368104 26136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:50.368130 26131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:50.369258 26121 server_base.cc:1034] running on GCE node
I20260809 00:53:50.370131 26121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:50.371246 26121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:50.372542 26121 hybrid_clock.cc:648] HybridClock initialized: now 1786236830372465 us; error 63 us; skew 500 ppm
I20260809 00:53:50.372977 26121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:53:50.375870 26121 webserver.cc:460] Webserver started at http://127.9.225.1:40855/ using document root <none> and password file <none>
I20260809 00:53:50.376730 26121 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:50.376886 26121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:50.377271 26121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:50.380687 26121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "bbe8148fcfcd41b5b56eaa757455609d"
format_stamp: "Formatted at 2026-08-09 00:53:50 on dist-test-slave-j2vl"
server_key: "489292801b8d4c393b5166ae51494cb0"
server_key_iv: "bb994a071b5463b9c5397fb568d78315"
server_key_version: "encryptionkey@0"
I20260809 00:53:50.381673 26121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "bbe8148fcfcd41b5b56eaa757455609d"
format_stamp: "Formatted at 2026-08-09 00:53:50 on dist-test-slave-j2vl"
server_key: "489292801b8d4c393b5166ae51494cb0"
server_key_iv: "bb994a071b5463b9c5397fb568d78315"
server_key_version: "encryptionkey@0"
I20260809 00:53:50.387768 26121 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260809 00:53:50.392380 26146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:50.394129 26121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 00:53:50.394353 26121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "bbe8148fcfcd41b5b56eaa757455609d"
format_stamp: "Formatted at 2026-08-09 00:53:50 on dist-test-slave-j2vl"
server_key: "489292801b8d4c393b5166ae51494cb0"
server_key_iv: "bb994a071b5463b9c5397fb568d78315"
server_key_version: "encryptionkey@0"
I20260809 00:53:50.394639 26121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:50.401491 26121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:50.402534 26121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:50.402863 26121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:50.404407 26121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:50.406345 26121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:50.406488 26121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:50.406641 26121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:50.406733 26121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:50.443156 26121 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:46853
I20260809 00:53:50.443197 26334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:46853 every 8 connection(s)
I20260809 00:53:50.444784 26121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:53:50.452209 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 26121
I20260809 00:53:50.452457 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:53:50.453248 10116 external_mini_cluster.cc:1428] Setting key 62b8b8aa31a76613117b4c847b63669a
I20260809 00:53:50.454545 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:50.447674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:52717 (local address 127.9.225.62:36775)
0809 00:53:50.448267 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:50.448276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:50.449724 (+ 1448us) server_negotiation.cc:407] Connection header received
0809 00:53:50.451665 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:50.451671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:50.451757 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:50.451902 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:50.453590 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:50.453597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:50.453602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:50.453628 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:50.453686 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:50.453706 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:50.453853 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:50.454083 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 00:53:50.454168 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":294,"threads_started":1}
I20260809 00:53:50.455591 26342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:50.448346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:36775 (local address 127.9.225.1:52717)
0809 00:53:50.449559 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:50.449608 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:50.451350 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:50.452130 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:50.452140 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:50.452246 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:50.453131 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:53:50.453134 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:50.453180 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:53:50.453285 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:50.453817 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:50.453826 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:50.453832 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 00:53:50.454174 (+ 342us) client_negotiation.cc:240] Negotiation successful
0809 00:53:50.454492 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":275,"threads_started":1}
I20260809 00:53:50.456578 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:36775
--builtin_ntp_servers=127.9.225.20:41521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 00:53:50.457393 26335 heartbeater.cc:347] Connected to a master server at 127.9.225.62:36775
I20260809 00:53:50.457769 26335 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:50.458601 26335 heartbeater.cc:511] Master 127.9.225.62:36775 requested a full tablet report, sending...
I20260809 00:53:50.461479 26014 ts_manager.cc:194] Registered new tserver with Master: bbe8148fcfcd41b5b56eaa757455609d (127.9.225.1:46853)
W20260809 00:53:50.840435 26343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:50.840790 26343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:50.840945 26343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:50.849193 26343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:50.849424 26343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:50.849607 26343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:53:50.859896 26343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:36775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:50.861469 26343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:50.863587 26343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:50.879624 26351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:50.879742 26350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:50.879781 26354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:50.880558 26343 server_base.cc:1034] running on GCE node
I20260809 00:53:50.881421 26343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:50.882575 26343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:50.883836 26343 hybrid_clock.cc:648] HybridClock initialized: now 1786236830883786 us; error 49 us; skew 500 ppm
I20260809 00:53:50.884214 26343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:53:50.886885 26343 webserver.cc:460] Webserver started at http://127.9.225.2:39713/ using document root <none> and password file <none>
I20260809 00:53:50.887722 26343 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:50.887878 26343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:50.888252 26343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:50.891780 26343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "c48e2a36f0bc49fd858acc3a135a9be9"
format_stamp: "Formatted at 2026-08-09 00:53:50 on dist-test-slave-j2vl"
server_key: "cb8e43ca8b17f79796d5ea5b0175bb55"
server_key_iv: "ae002ea0447bd528971c4a9610584a5f"
server_key_version: "encryptionkey@0"
I20260809 00:53:50.892757 26343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "c48e2a36f0bc49fd858acc3a135a9be9"
format_stamp: "Formatted at 2026-08-09 00:53:50 on dist-test-slave-j2vl"
server_key: "cb8e43ca8b17f79796d5ea5b0175bb55"
server_key_iv: "ae002ea0447bd528971c4a9610584a5f"
server_key_version: "encryptionkey@0"
I20260809 00:53:50.899178 26343 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260809 00:53:50.903429 26363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:50.905324 26343 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260809 00:53:50.905659 26343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "c48e2a36f0bc49fd858acc3a135a9be9"
format_stamp: "Formatted at 2026-08-09 00:53:50 on dist-test-slave-j2vl"
server_key: "cb8e43ca8b17f79796d5ea5b0175bb55"
server_key_iv: "ae002ea0447bd528971c4a9610584a5f"
server_key_version: "encryptionkey@0"
I20260809 00:53:50.905961 26343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:50.912310 26343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:50.913412 26343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:50.913749 26343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:50.914870 26343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:50.916472 26343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:50.916605 26343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:50.916757 26343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:50.916850 26343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:50.952824 26343 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:35135
I20260809 00:53:50.952879 26539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:35135 every 8 connection(s)
I20260809 00:53:50.954378 26343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:53:50.963869 26543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:50.957493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:33261 (local address 127.9.225.62:36775)
0809 00:53:50.957972 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:50.957977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:50.959285 (+ 1308us) server_negotiation.cc:407] Connection header received
0809 00:53:50.961144 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:50.961150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:50.961236 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:50.961386 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:50.962983 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:50.962992 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:50.963001 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:50.963022 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:50.963073 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:50.963089 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:50.963245 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:50.963452 (+ 207us) server_negotiation.cc:299] Negotiation successful
0809 00:53:50.963537 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":249,"threads_started":1}
I20260809 00:53:50.964088 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 26343
I20260809 00:53:50.964304 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:53:50.964764 10116 external_mini_cluster.cc:1428] Setting key e1a469e0a13dddbdbcffc0712b5f917f
I20260809 00:53:50.965153 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:50.958071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:36775 (local address 127.9.225.2:33261)
0809 00:53:50.959097 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:50.959137 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:50.960815 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:50.961666 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:50.961676 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:50.961783 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:50.962584 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:53:50.962588 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:50.962649 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:53:50.962777 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:50.963196 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:50.963208 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:50.963219 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 00:53:50.963545 (+ 326us) client_negotiation.cc:240] Negotiation successful
0809 00:53:50.963861 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":291,"threads_started":1}
I20260809 00:53:50.966794 26540 heartbeater.cc:347] Connected to a master server at 127.9.225.62:36775
I20260809 00:53:50.967196 26540 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:50.967870 26540 heartbeater.cc:511] Master 127.9.225.62:36775 requested a full tablet report, sending...
I20260809 00:53:50.968053 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:36775
--builtin_ntp_servers=127.9.225.20:41521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 00:53:50.969655 26014 ts_manager.cc:194] Registered new tserver with Master: c48e2a36f0bc49fd858acc3a135a9be9 (127.9.225.2:35135)
W20260809 00:53:51.348639 26545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:51.349058 26545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:51.349218 26545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:51.358368 26545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:51.358528 26545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:51.358738 26545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:53:51.369335 26545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:36775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:51.371018 26545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:51.373248 26545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:51.388691 26555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:51.388626 26553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:51.388593 26552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:51.390178 26545 server_base.cc:1034] running on GCE node
I20260809 00:53:51.391000 26545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:51.392204 26545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:51.393515 26545 hybrid_clock.cc:648] HybridClock initialized: now 1786236831393474 us; error 46 us; skew 500 ppm
I20260809 00:53:51.393899 26545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:53:51.396649 26545 webserver.cc:460] Webserver started at http://127.9.225.3:39401/ using document root <none> and password file <none>
I20260809 00:53:51.397531 26545 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:51.397640 26545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:51.397955 26545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:51.401424 26545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "c0158dfa197e42928c17667e3e095410"
format_stamp: "Formatted at 2026-08-09 00:53:51 on dist-test-slave-j2vl"
server_key: "b482ea88a8491a1a05d7b2c5ca1fe4eb"
server_key_iv: "d605a29fe121ccadbd0553712bdbbb02"
server_key_version: "encryptionkey@0"
I20260809 00:53:51.402462 26545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "c0158dfa197e42928c17667e3e095410"
format_stamp: "Formatted at 2026-08-09 00:53:51 on dist-test-slave-j2vl"
server_key: "b482ea88a8491a1a05d7b2c5ca1fe4eb"
server_key_iv: "d605a29fe121ccadbd0553712bdbbb02"
server_key_version: "encryptionkey@0"
I20260809 00:53:51.408391 26545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 00:53:51.412751 26563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:51.414765 26545 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:53:51.415006 26545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "c0158dfa197e42928c17667e3e095410"
format_stamp: "Formatted at 2026-08-09 00:53:51 on dist-test-slave-j2vl"
server_key: "b482ea88a8491a1a05d7b2c5ca1fe4eb"
server_key_iv: "d605a29fe121ccadbd0553712bdbbb02"
server_key_version: "encryptionkey@0"
I20260809 00:53:51.415285 26545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:51.422042 26545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:51.423058 26545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:51.423379 26545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:51.424538 26545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:51.426586 26545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:51.426725 26545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:51.426870 26545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:51.426980 26545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:51.464104 26545 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:36885
I20260809 00:53:51.464210 26733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:36885 every 8 connection(s)
I20260809 00:53:51.465734 26545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:53:51.466272 26335 heartbeater.cc:503] Master 127.9.225.62:36775 was elected leader, sending a full tablet report...
I20260809 00:53:51.472236 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 26545
I20260809 00:53:51.472536 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:53:51.473059 10116 external_mini_cluster.cc:1428] Setting key 9ea8c0a2826330302ffd98efe035cec1
I20260809 00:53:51.475970 26737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:51.469008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:33721 (local address 127.9.225.62:36775)
0809 00:53:51.469528 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:51.469534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:51.471037 (+ 1503us) server_negotiation.cc:407] Connection header received
0809 00:53:51.473072 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:51.473081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:51.473168 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:51.473312 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:51.475011 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:51.475018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:51.475023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:51.475048 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:51.475112 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:51.475139 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:51.475290 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:51.475540 (+ 250us) server_negotiation.cc:299] Negotiation successful
0809 00:53:51.475615 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":234,"threads_started":1}
I20260809 00:53:51.477314 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:51.469741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:36775 (local address 127.9.225.3:33721)
0809 00:53:51.470816 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:51.470860 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:51.472708 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:51.473664 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:51.473677 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:51.473823 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:51.474645 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:53:51.474648 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:51.474695 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:53:51.474815 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:51.475249 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:51.475258 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:51.475264 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 00:53:51.475611 (+ 347us) client_negotiation.cc:240] Negotiation successful
0809 00:53:51.475990 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":315,"threads_started":1}
I20260809 00:53:51.478837 26735 heartbeater.cc:347] Connected to a master server at 127.9.225.62:36775
I20260809 00:53:51.479146 26735 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:51.479760 26735 heartbeater.cc:511] Master 127.9.225.62:36775 requested a full tablet report, sending...
I20260809 00:53:51.481817 26014 ts_manager.cc:194] Registered new tserver with Master: c0158dfa197e42928c17667e3e095410 (127.9.225.3:36885)
I20260809 00:53:51.490480 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 00:53:51.498032 26737 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:53:51.496043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.9.225.62:36775)
0809 00:53:51.496285 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:51.496294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:51.496843 (+ 549us) server_negotiation.cc:407] Connection header received
0809 00:53:51.497071 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:51.497076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:51.497139 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:51.497237 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:51.497732 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53442: BlockingRecv error: recv got EOF from 127.0.0.1:53442 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 00:53:51.498873 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 26121
I20260809 00:53:51.503525 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:51.500513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.9.225.1:46853)
0809 00:53:51.501206 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:51.501214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:51.501242 (+ 28us) server_negotiation.cc:407] Connection header received
0809 00:53:51.501310 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:51.501319 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:51.501634 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:51.501780 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:51.502811 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:51.502826 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:51.502840 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:51.502853 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:51.502897 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:51.502912 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:51.503054 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:51.503134 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 00:53:51.503224 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":373,"threads_started":1}
I20260809 00:53:51.504909 26327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:51.592985 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 26121
I20260809 00:53:51.625771 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 26343
I20260809 00:53:51.629791 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:51.626848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.9.225.2:35135)
0809 00:53:51.627373 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:51.627381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:51.627417 (+ 36us) server_negotiation.cc:407] Connection header received
0809 00:53:51.627494 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:51.627501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:51.627740 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:51.627924 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:51.628983 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:51.628990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:51.629002 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:51.629014 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:51.629069 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:51.629084 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:51.629243 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:51.629374 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 00:53:51.629508 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":224,"threads_started":1}
I20260809 00:53:51.631134 26530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:51.717527 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 26343
I20260809 00:53:51.748310 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 26545
I20260809 00:53:51.752960 26754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:51.749422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.9.225.3:36885)
0809 00:53:51.750138 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:51.750147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:51.750183 (+ 36us) server_negotiation.cc:407] Connection header received
0809 00:53:51.750279 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:51.750287 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:51.750547 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:51.750749 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:51.752087 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:51.752094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:51.752108 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:53:51.752122 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:51.752169 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:53:51.752188 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:51.752409 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:51.752490 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 00:53:51.752621 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":331,"threads_started":1}
I20260809 00:53:51.754637 26729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:51.842437 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 26545
I20260809 00:53:51.871600 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 25958
I20260809 00:53:51.873013 26064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:51.983448 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 25958
2026-08-09T00:53:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2814 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:53:52 dist-test-slave-j2vl krb5kdc[26760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:53:52 dist-test-slave-j2vl krb5kdc[26760](info): set up 2 sockets
Aug 09 00:53:52 dist-test-slave-j2vl krb5kdc[26760](info): commencing operation
krb5kdc: starting...
W20260809 00:53:53.411372 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.372s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:53:53 dist-test-slave-j2vl krb5kdc[26760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:53:53.757660 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35955
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:40881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:35955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:54.172492 26778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:54.172904 26778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:54.173079 26778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:54.181619 26778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:53:54.181829 26778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:54.181952 26778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:54.182082 26778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:53:54.182164 26778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:53:54.193570 26778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:40881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:35955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:54.195273 26778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:54.196976 26778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:54.210650 26784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:54.211553 26789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:54.211630 26793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:54.212042 26778 server_base.cc:1034] running on GCE node
I20260809 00:53:54.212996 26778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:54.214797 26778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:54.216051 26778 hybrid_clock.cc:648] HybridClock initialized: now 1786236834215992 us; error 39 us; skew 500 ppm
I20260809 00:53:54.216485 26778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:54 dist-test-slave-j2vl krb5kdc[26760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236834, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:54.219911 26778 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:53:54.221738 26778 webserver.cc:460] Webserver started at http://127.9.225.62:33589/ using document root <none> and password file <none>
I20260809 00:53:54.222563 26778 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:54.222720 26778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:54.223083 26778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:54.226560 26778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "ba378f1b372e41468c3053f63d5a1cdd"
format_stamp: "Formatted at 2026-08-09 00:53:54 on dist-test-slave-j2vl"
server_key: "b30bdfc1905b408aa1e0b4c4dfd23101"
server_key_iv: "2cd8985533512ef74d386897ef678586"
server_key_version: "encryptionkey@0"
I20260809 00:53:54.227470 26778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "ba378f1b372e41468c3053f63d5a1cdd"
format_stamp: "Formatted at 2026-08-09 00:53:54 on dist-test-slave-j2vl"
server_key: "b30bdfc1905b408aa1e0b4c4dfd23101"
server_key_iv: "2cd8985533512ef74d386897ef678586"
server_key_version: "encryptionkey@0"
I20260809 00:53:54.233759 26778 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 00:53:54.238528 26803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:54.240352 26778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:53:54.240629 26778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "ba378f1b372e41468c3053f63d5a1cdd"
format_stamp: "Formatted at 2026-08-09 00:53:54 on dist-test-slave-j2vl"
server_key: "b30bdfc1905b408aa1e0b4c4dfd23101"
server_key_iv: "2cd8985533512ef74d386897ef678586"
server_key_version: "encryptionkey@0"
I20260809 00:53:54.240893 26778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:54.263520 26778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:54.268996 26778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:54.269397 26778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:54.291613 26778 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:35955
I20260809 00:53:54.291646 26885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:35955 every 8 connection(s)
I20260809 00:53:54.293246 26778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:53:54.297995 26886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:54.299453 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 26778
I20260809 00:53:54.299746 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:53:54.300438 10116 external_mini_cluster.cc:1428] Setting key 9921f5ebba716aa08bca9eeef5f81b2b
I20260809 00:53:54.314481 26886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd: Bootstrap starting.
Aug 09 00:53:54 dist-test-slave-j2vl krb5kdc[26760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:53:54.318903 26886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:54.320189 26886 log.cc:824] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:54.324221 26886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd: No bootstrap required, opened a new log
I20260809 00:53:54.328660 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:54.302945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.9.225.62:35955)
0809 00:53:54.303967 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:54.303982 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:54.304035 (+ 53us) server_negotiation.cc:407] Connection header received
0809 00:53:54.305854 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:54.305889 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:54.306381 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:54.306772 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:54.308964 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:54.310490 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:54.312479 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:54.313260 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:54.313515 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:54.318757 (+ 5242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:54.318768 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:54.318790 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:54.318844 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:54.322992 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:54.323989 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:54.324000 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:54.324010 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:54.324286 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:54.324805 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:54.324812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:54.324817 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:54.325708 (+ 891us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:54.326071 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:54.326506 (+ 435us) server_negotiation.cc:299] Negotiation successful
0809 00:53:54.327049 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":745,"thread_start_us":476,"threads_started":1}
I20260809 00:53:54.334627 26886 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba378f1b372e41468c3053f63d5a1cdd" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35955 } }
I20260809 00:53:54.335091 26886 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:54.335175 26886 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba378f1b372e41468c3053f63d5a1cdd, State: Initialized, Role: FOLLOWER
I20260809 00:53:54.335830 26886 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba378f1b372e41468c3053f63d5a1cdd" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35955 } }
I20260809 00:53:54.336110 26886 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:53:54.336277 26886 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:53:54.336472 26886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:54.339191 26886 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba378f1b372e41468c3053f63d5a1cdd" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35955 } }
I20260809 00:53:54.339824 26886 leader_election.cc:302] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba378f1b372e41468c3053f63d5a1cdd; no voters:
I20260809 00:53:54.340595 26886 leader_election.cc:288] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:53:54.340731 26892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:54.341521 26892 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [term 1 LEADER]: Becoming Leader. State: Replica: ba378f1b372e41468c3053f63d5a1cdd, State: Running, Role: LEADER
I20260809 00:53:54.342193 26892 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba378f1b372e41468c3053f63d5a1cdd" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35955 } }
I20260809 00:53:54.343396 26886 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:53:54.346376 26894 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba378f1b372e41468c3053f63d5a1cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba378f1b372e41468c3053f63d5a1cdd" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35955 } } }
I20260809 00:53:54.346869 26894 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:54.348001 26895 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba378f1b372e41468c3053f63d5a1cdd. Latest consensus state: current_term: 1 leader_uuid: "ba378f1b372e41468c3053f63d5a1cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba378f1b372e41468c3053f63d5a1cdd" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35955 } } }
I20260809 00:53:54.348539 26895 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd [sys.catalog]: This master's current role is: LEADER
I20260809 00:53:54.351207 26900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:53:54.358224 26900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:53:54.369068 26900 catalog_manager.cc:1347] Generated new cluster ID: bd7680ef9ce749719a9f545925f94a69
I20260809 00:53:54.369231 26900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:53:54.395673 26900 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:53:54.397265 26900 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:53:54.410895 26900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd: Generated new TSK 0
I20260809 00:53:54.412086 26900 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:53:54.502298 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35955
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:40881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:54.903584 26930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:54.903959 26930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:54.904068 26930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:54.912115 26930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:54.912259 26930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:54.912384 26930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:53:54.923347 26930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:40881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:54.924921 26930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:54.926883 26930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:54.943157 26938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:54.943123 26942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:54.943089 26946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:54.944166 26930 server_base.cc:1034] running on GCE node
I20260809 00:53:54.945073 26930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:54.946476 26930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:54.947737 26930 hybrid_clock.cc:648] HybridClock initialized: now 1786236834947691 us; error 53 us; skew 500 ppm
I20260809 00:53:54.948145 26930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:54 dist-test-slave-j2vl krb5kdc[26760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236834, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:54.955595 26930 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:53:54.957590 26930 webserver.cc:460] Webserver started at http://127.9.225.1:45739/ using document root <none> and password file <none>
I20260809 00:53:54.958475 26930 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:54.958659 26930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:54.959024 26930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:54.962679 26930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa"
format_stamp: "Formatted at 2026-08-09 00:53:54 on dist-test-slave-j2vl"
server_key: "431c8c5d9e3378ce343921f52f9fbf5c"
server_key_iv: "3056a0a18a55f06fbf6b8c8895130118"
server_key_version: "encryptionkey@0"
I20260809 00:53:54.963588 26930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa"
format_stamp: "Formatted at 2026-08-09 00:53:54 on dist-test-slave-j2vl"
server_key: "431c8c5d9e3378ce343921f52f9fbf5c"
server_key_iv: "3056a0a18a55f06fbf6b8c8895130118"
server_key_version: "encryptionkey@0"
I20260809 00:53:54.969913 26930 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 00:53:54.974264 26958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:54.976039 26930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:53:54.976308 26930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa"
format_stamp: "Formatted at 2026-08-09 00:53:54 on dist-test-slave-j2vl"
server_key: "431c8c5d9e3378ce343921f52f9fbf5c"
server_key_iv: "3056a0a18a55f06fbf6b8c8895130118"
server_key_version: "encryptionkey@0"
I20260809 00:53:54.976579 26930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:54.994282 26930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:55.000324 26930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:55.000721 26930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:55.002331 26930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:55.004175 26930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:55.004320 26930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:55.004490 26930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:55.004573 26930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:55.044195 26930 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:40149
I20260809 00:53:55.044214 27127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:40149 every 8 connection(s)
I20260809 00:53:55.045809 26930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:53:55.048801 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 26930
I20260809 00:53:55.049084 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:53:55.050005 10116 external_mini_cluster.cc:1428] Setting key 6936a677b41952e41e130bdf05b59576
Aug 09 00:53:55 dist-test-slave-j2vl krb5kdc[26760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236834, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:53:55.072283 27131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:55.049089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:45611 (local address 127.9.225.62:35955)
0809 00:53:55.049703 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:55.049710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:55.050997 (+ 1287us) server_negotiation.cc:407] Connection header received
0809 00:53:55.053275 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:55.053284 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:55.053410 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:55.053538 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:55.055345 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:55.056565 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:55.058529 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:55.059282 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:55.059447 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:55.063729 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:55.063741 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:55.063749 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:55.063806 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:55.067482 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:55.068733 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:55.068739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:55.068741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:55.068896 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:55.069714 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:55.069719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:55.069721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:55.070086 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:55.070247 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:55.071713 (+ 1466us) server_negotiation.cc:299] Negotiation successful
0809 00:53:55.071985 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":317,"threads_started":1}
I20260809 00:53:55.074146 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:55.049807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35955 (local address 127.9.225.1:45611)
0809 00:53:55.050796 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:55.050849 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:55.052909 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:55.053777 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:55.053787 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:55.054418 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:55.055112 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:55.055134 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:55.056799 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:55.056804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:55.058328 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:55.058344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:55.059554 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:55.059560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:55.059796 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:55.060643 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:55.060672 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:55.063379 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:55.067718 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:55.067729 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:55.067756 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:55.068536 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:55.069029 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:55.069037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:55.069044 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:55.069333 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:55.070305 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:55.070323 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:55.071125 (+ 802us) client_negotiation.cc:769] Sending connection context
0809 00:53:55.071584 (+ 459us) client_negotiation.cc:240] Negotiation successful
0809 00:53:55.072325 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":443,"threads_started":1}
I20260809 00:53:55.076361 27128 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35955
I20260809 00:53:55.076911 27128 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:55.077950 27128 heartbeater.cc:511] Master 127.9.225.62:35955 requested a full tablet report, sending...
I20260809 00:53:55.081121 26832 ts_manager.cc:194] Registered new tserver with Master: 4ecbe7e05f1d49beb618ec597bf3c4fa (127.9.225.1:40149)
I20260809 00:53:55.084395 26832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:45611
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:53:55.135246 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35955
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:40881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:53:55.528210 27137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:55.528563 27137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:55.528759 27137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:55.536479 27137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:55.536628 27137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:55.536832 27137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:53:55.547742 27137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:40881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:55.549321 27137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:55.550865 27137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:55.566349 27149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:55.566421 27152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:55.566329 27154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:55.567482 27137 server_base.cc:1034] running on GCE node
I20260809 00:53:55.568313 27137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:55.569609 27137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:55.570887 27137 hybrid_clock.cc:648] HybridClock initialized: now 1786236835570836 us; error 51 us; skew 500 ppm
I20260809 00:53:55.571321 27137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:55 dist-test-slave-j2vl krb5kdc[26760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236835, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:55.575055 27137 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:53:55.576921 27137 webserver.cc:460] Webserver started at http://127.9.225.2:36233/ using document root <none> and password file <none>
I20260809 00:53:55.577850 27137 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:55.577979 27137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:55.578295 27137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:55.582032 27137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "66bac660662a4158918a9d574e889e09"
format_stamp: "Formatted at 2026-08-09 00:53:55 on dist-test-slave-j2vl"
server_key: "0c919f4b2830e4d065eca1c5fc3c49d9"
server_key_iv: "fc16151a143f61b0fa2ca52d682ac74f"
server_key_version: "encryptionkey@0"
I20260809 00:53:55.582960 27137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "66bac660662a4158918a9d574e889e09"
format_stamp: "Formatted at 2026-08-09 00:53:55 on dist-test-slave-j2vl"
server_key: "0c919f4b2830e4d065eca1c5fc3c49d9"
server_key_iv: "fc16151a143f61b0fa2ca52d682ac74f"
server_key_version: "encryptionkey@0"
I20260809 00:53:55.589644 27137 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 00:53:55.593918 27164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:55.595724 27137 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:53:55.595918 27137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "66bac660662a4158918a9d574e889e09"
format_stamp: "Formatted at 2026-08-09 00:53:55 on dist-test-slave-j2vl"
server_key: "0c919f4b2830e4d065eca1c5fc3c49d9"
server_key_iv: "fc16151a143f61b0fa2ca52d682ac74f"
server_key_version: "encryptionkey@0"
I20260809 00:53:55.596144 27137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:55.611768 27137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:55.617148 27137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:55.617573 27137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:55.618721 27137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:55.620350 27137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:55.620436 27137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:55.620533 27137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:55.620585 27137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:55.659535 27137 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:43829
I20260809 00:53:55.659577 27351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:43829 every 8 connection(s)
I20260809 00:53:55.661033 27137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:53:55.663100 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 27137
I20260809 00:53:55.663383 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:53:55.663996 10116 external_mini_cluster.cc:1428] Setting key 26bbb561021acefa4fc68befd61663f3
Aug 09 00:53:55 dist-test-slave-j2vl krb5kdc[26760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236835, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:53:55.687331 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:55.664740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:48919 (local address 127.9.225.62:35955)
0809 00:53:55.665411 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:55.665418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:55.666377 (+ 959us) server_negotiation.cc:407] Connection header received
0809 00:53:55.668355 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:55.668361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:55.668454 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:55.668595 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:55.670404 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:55.671178 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:55.672883 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:55.673404 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:55.673555 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:55.678387 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:55.678394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:55.678398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:55.678445 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:55.681959 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:55.683301 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:55.683315 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:55.683326 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:55.683609 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:55.684361 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:55.684366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:55.684368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:55.685022 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:55.685384 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:55.686460 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0809 00:53:55.686788 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":329,"threads_started":1}
I20260809 00:53:55.688583 27359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:55.665103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35955 (local address 127.9.225.2:48919)
0809 00:53:55.666148 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:55.666199 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:55.668051 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:55.668869 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:55.668878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:55.669553 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:55.670207 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:55.670226 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:55.671362 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:55.671367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:55.672695 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:55.672710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:55.673696 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:55.673704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:55.674013 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:55.674847 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:55.674876 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:55.677990 (+ 3114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:55.682198 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:55.682211 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:55.682236 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:55.682908 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:55.683788 (+ 880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:55.683797 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:55.683806 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:55.684179 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:55.685328 (+ 1149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:55.685399 (+ 71us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:55.685924 (+ 525us) client_negotiation.cc:769] Sending connection context
0809 00:53:55.686289 (+ 365us) client_negotiation.cc:240] Negotiation successful
0809 00:53:55.686979 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":527,"threads_started":1}
I20260809 00:53:55.690659 27352 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35955
I20260809 00:53:55.691082 27352 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:55.691984 27352 heartbeater.cc:511] Master 127.9.225.62:35955 requested a full tablet report, sending...
I20260809 00:53:55.694011 26832 ts_manager.cc:194] Registered new tserver with Master: 66bac660662a4158918a9d574e889e09 (127.9.225.2:43829)
I20260809 00:53:55.695047 26832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:48919
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:53:55.742355 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35955
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:40881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:53:56.088614 27128 heartbeater.cc:503] Master 127.9.225.62:35955 was elected leader, sending a full tablet report...
W20260809 00:53:56.144415 27364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:53:56.144881 27364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:53:56.145066 27364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:53:56.152819 27364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:53:56.153025 27364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:53:56.153244 27364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:53:56.164258 27364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:40881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:53:56.166045 27364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:53:56.167527 27364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:53:56.182610 27378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:56.182569 27374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:53:56.182538 27375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:53:56.183476 27364 server_base.cc:1034] running on GCE node
I20260809 00:53:56.184288 27364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:53:56.185469 27364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:53:56.186719 27364 hybrid_clock.cc:648] HybridClock initialized: now 1786236836186648 us; error 66 us; skew 500 ppm
I20260809 00:53:56.187119 27364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:53:56 dist-test-slave-j2vl krb5kdc[26760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236836, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:53:56.191298 27364 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:53:56.193156 27364 webserver.cc:460] Webserver started at http://127.9.225.3:35177/ using document root <none> and password file <none>
I20260809 00:53:56.194105 27364 fs_manager.cc:359] Metadata directory not provided
I20260809 00:53:56.194278 27364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:53:56.194646 27364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:53:56.198258 27364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2"
format_stamp: "Formatted at 2026-08-09 00:53:56 on dist-test-slave-j2vl"
server_key: "d5eaf8b75c181b65280fb749be2ac0bd"
server_key_iv: "61456741820016c62193bd3ec3e7629f"
server_key_version: "encryptionkey@0"
I20260809 00:53:56.199203 27364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2"
format_stamp: "Formatted at 2026-08-09 00:53:56 on dist-test-slave-j2vl"
server_key: "d5eaf8b75c181b65280fb749be2ac0bd"
server_key_iv: "61456741820016c62193bd3ec3e7629f"
server_key_version: "encryptionkey@0"
I20260809 00:53:56.206085 27364 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 00:53:56.210882 27387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:56.212692 27364 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:53:56.212965 27364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2"
format_stamp: "Formatted at 2026-08-09 00:53:56 on dist-test-slave-j2vl"
server_key: "d5eaf8b75c181b65280fb749be2ac0bd"
server_key_iv: "61456741820016c62193bd3ec3e7629f"
server_key_version: "encryptionkey@0"
I20260809 00:53:56.213195 27364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:53:56.243156 27364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:53:56.249673 27364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:53:56.250000 27364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:53:56.251060 27364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:53:56.252918 27364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:53:56.253051 27364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:56.253180 27364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:53:56.253228 27364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:53:56.292899 27364 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:34647
I20260809 00:53:56.292899 27581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:34647 every 8 connection(s)
I20260809 00:53:56.294930 27364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:53:56.298812 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 27364
I20260809 00:53:56.299101 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:53:56.299774 10116 external_mini_cluster.cc:1428] Setting key ffc0d29d7632314f02259d639400ea97
Aug 09 00:53:56 dist-test-slave-j2vl krb5kdc[26760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236836, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:53:56.325336 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.298298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:45467 (local address 127.9.225.62:35955)
0809 00:53:56.306106 (+ 7808us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:56.306111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:56.306129 (+ 18us) server_negotiation.cc:407] Connection header received
0809 00:53:56.307855 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:56.307860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:56.307949 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:56.308064 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:56.309970 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.310862 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.312566 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.313014 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.313140 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.317514 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:56.317521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:56.317526 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:56.317573 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:56.321165 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:56.322222 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:56.322227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:56.322230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:56.322390 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:56.323264 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:56.323269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:56.323271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:56.323636 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:56.323783 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:56.324720 (+ 937us) server_negotiation.cc:299] Negotiation successful
0809 00:53:56.324977 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7693,"thread_start_us":7565,"threads_started":1}
I20260809 00:53:56.326489 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.299037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35955 (local address 127.9.225.3:45467)
0809 00:53:56.305489 (+ 6452us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:56.305562 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:56.307486 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:56.308329 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:56.308338 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:56.308988 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:53:56.309758 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:56.309778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.311030 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.311033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:56.312375 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:56.312390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.313248 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.313253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:56.313550 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.314340 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:53:56.314366 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:53:56.317227 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:53:56.321440 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:56.321460 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:56.321478 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:56.322062 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:56.322622 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:53:56.322628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:53:56.322631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:53:56.323079 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:53:56.323825 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:53:56.323833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:53:56.324288 (+ 455us) client_negotiation.cc:769] Sending connection context
0809 00:53:56.324587 (+ 299us) client_negotiation.cc:240] Negotiation successful
0809 00:53:56.325095 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6049,"thread_start_us":5762,"threads_started":1}
I20260809 00:53:56.328313 27582 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35955
I20260809 00:53:56.328719 27582 heartbeater.cc:464] Registering TS with master...
I20260809 00:53:56.329923 27582 heartbeater.cc:511] Master 127.9.225.62:35955 requested a full tablet report, sending...
I20260809 00:53:56.331921 26832 ts_manager.cc:194] Registered new tserver with Master: e0cbd260f7424ab386ae3e6b75ae2aa2 (127.9.225.3:34647)
I20260809 00:53:56.332844 26832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:45467
I20260809 00:53:56.344460 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:53:56.363914 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.349759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.9.225.62:35955)
0809 00:53:56.349956 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:56.349961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:56.350585 (+ 624us) server_negotiation.cc:407] Connection header received
0809 00:53:56.350730 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:56.350734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:56.350821 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:56.350924 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:56.352586 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.353435 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.354878 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.355348 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.355477 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.358140 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:56.358146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:56.358150 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:56.358189 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:56.361052 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:56.361916 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:56.361921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:56.361923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:56.362056 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:56.362440 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:56.362445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:56.362447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:56.362854 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:56.363067 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:56.363451 (+ 384us) server_negotiation.cc:299] Negotiation successful
0809 00:53:56.363650 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 00:53:56.368656 26832 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:53:56.376296 26832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:53:56.419667 27603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.391835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:40149 (local address 127.0.0.1:54622)
0809 00:53:56.407809 (+ 15974us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:56.407893 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:56.408213 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:56.410874 (+ 2661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:56.410884 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:56.410938 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:56.411459 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:56.411473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.414196 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.414201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:56.416476 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:56.416490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.418474 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.418482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:56.418689 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.418767 (+ 78us) client_negotiation.cc:769] Sending connection context
0809 00:53:56.418917 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 00:53:56.419164 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15473,"mutex_wait_us":57,"thread_start_us":19924,"threads_started":1}
I20260809 00:53:56.420572 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.392134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:43829 (local address 127.0.0.1:37544)
0809 00:53:56.410875 (+ 18741us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:56.410895 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:56.411020 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:56.412068 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:56.412075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:56.412107 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:56.412540 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:56.412551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.414594 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.414599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:56.417186 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:56.417202 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.419586 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.419595 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:56.419838 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.419877 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 00:53:56.419978 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 00:53:56.420124 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18584,"mutex_wait_us":27,"thread_start_us":18408,"threads_started":1}
I20260809 00:53:56.420939 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.391996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.9.225.1:40149)
0809 00:53:56.410023 (+ 18027us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:56.410035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:56.410079 (+ 44us) server_negotiation.cc:407] Connection header received
0809 00:53:56.410220 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:56.410229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:56.410536 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:56.411783 (+ 1247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:56.412090 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.414031 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.416659 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.418218 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.420036 (+ 1818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.420289 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:56.420369 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 00:53:56.420571 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17814,"thread_start_us":17657,"threads_started":1}
I20260809 00:53:56.421208 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.392578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:34647 (local address 127.0.0.1:37492)
0809 00:53:56.412096 (+ 19518us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:56.412114 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:56.412263 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:56.412860 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:56.412867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:56.412894 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:56.413241 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:56.413256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.415347 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.415352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:56.417641 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:56.417652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.419892 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.419901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:56.420076 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.420115 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 00:53:56.420223 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 00:53:56.420378 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19383,"mutex_wait_us":4,"thread_start_us":15955,"threads_started":1}
I20260809 00:53:56.421478 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.393418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.9.225.3:34647)
0809 00:53:56.400966 (+ 7548us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:56.400980 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:56.412267 (+ 11287us) server_negotiation.cc:407] Connection header received
0809 00:53:56.412401 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:56.412409 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:56.412632 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:56.412844 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:56.413503 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.415141 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.417856 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.419366 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.419466 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.419629 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:56.420303 (+ 674us) server_negotiation.cc:299] Negotiation successful
0809 00:53:56.420509 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7245,"thread_start_us":6980,"threads_started":1}
I20260809 00:53:56.421689 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.392829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.9.225.2:43829)
0809 00:53:56.410346 (+ 17517us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:56.410354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:56.411171 (+ 817us) server_negotiation.cc:407] Connection header received
0809 00:53:56.411289 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:56.411298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:56.411571 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:56.411784 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:56.412705 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.414410 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.417431 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.419051 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.419178 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.419410 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:56.421071 (+ 1661us) server_negotiation.cc:299] Negotiation successful
0809 00:53:56.421263 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17362,"thread_start_us":17241,"threads_started":1}
I20260809 00:53:56.425300 27465 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd84fa84710491fa93dd2074d11e799 (DEFAULT_TABLE table=test-table [id=527831e5b539407e9ca53392ba42a470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:56.425307 27030 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd84fa84710491fa93dd2074d11e799 (DEFAULT_TABLE table=test-table [id=527831e5b539407e9ca53392ba42a470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:56.425300 27235 tablet_service.cc:1467] Processing CreateTablet for tablet 7bd84fa84710491fa93dd2074d11e799 (DEFAULT_TABLE table=test-table [id=527831e5b539407e9ca53392ba42a470]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:53:56.426910 27235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd84fa84710491fa93dd2074d11e799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:56.426997 27030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd84fa84710491fa93dd2074d11e799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:56.427016 27465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bd84fa84710491fa93dd2074d11e799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:53:56.440476 27610 tablet_bootstrap.cc:492] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09: Bootstrap starting.
I20260809 00:53:56.442740 27611 tablet_bootstrap.cc:492] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa: Bootstrap starting.
I20260809 00:53:56.443914 27613 tablet_bootstrap.cc:492] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2: Bootstrap starting.
I20260809 00:53:56.447014 27610 tablet_bootstrap.cc:654] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:56.447014 27611 tablet_bootstrap.cc:654] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:56.447613 27613 tablet_bootstrap.cc:654] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2: Neither blocks nor log segments found. Creating new log.
I20260809 00:53:56.448500 27610 log.cc:824] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:56.448500 27611 log.cc:824] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:56.448701 27613 log.cc:824] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2: Log is configured to *not* fsync() on all Append() calls
I20260809 00:53:56.452155 27613 tablet_bootstrap.cc:492] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2: No bootstrap required, opened a new log
I20260809 00:53:56.452261 27611 tablet_bootstrap.cc:492] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa: No bootstrap required, opened a new log
I20260809 00:53:56.452262 27610 tablet_bootstrap.cc:492] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09: No bootstrap required, opened a new log
I20260809 00:53:56.452525 27613 ts_tablet_manager.cc:1397] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260809 00:53:56.452554 27611 ts_tablet_manager.cc:1397] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260809 00:53:56.452553 27610 ts_tablet_manager.cc:1397] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260809 00:53:56.460659 27613 raft_consensus.cc:348] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40149 } } peers { permanent_uuid: "66bac660662a4158918a9d574e889e09" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43829 } } peers { permanent_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34647 } }
I20260809 00:53:56.461172 27613 raft_consensus.cc:374] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:56.461279 27613 raft_consensus.cc:729] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cbd260f7424ab386ae3e6b75ae2aa2, State: Initialized, Role: FOLLOWER
I20260809 00:53:56.462164 27613 consensus_queue.cc:260] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40149 } } peers { permanent_uuid: "66bac660662a4158918a9d574e889e09" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43829 } } peers { permanent_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34647 } }
I20260809 00:53:56.464278 27611 raft_consensus.cc:348] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40149 } } peers { permanent_uuid: "66bac660662a4158918a9d574e889e09" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43829 } } peers { permanent_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34647 } }
I20260809 00:53:56.464278 27610 raft_consensus.cc:348] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40149 } } peers { permanent_uuid: "66bac660662a4158918a9d574e889e09" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43829 } } peers { permanent_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34647 } }
I20260809 00:53:56.464797 27610 raft_consensus.cc:374] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:56.464799 27611 raft_consensus.cc:374] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:53:56.464903 27610 raft_consensus.cc:729] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bac660662a4158918a9d574e889e09, State: Initialized, Role: FOLLOWER
I20260809 00:53:56.464910 27611 raft_consensus.cc:729] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecbe7e05f1d49beb618ec597bf3c4fa, State: Initialized, Role: FOLLOWER
I20260809 00:53:56.465821 27610 consensus_queue.cc:260] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40149 } } peers { permanent_uuid: "66bac660662a4158918a9d574e889e09" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43829 } } peers { permanent_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34647 } }
I20260809 00:53:56.465821 27611 consensus_queue.cc:260] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40149 } } peers { permanent_uuid: "66bac660662a4158918a9d574e889e09" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43829 } } peers { permanent_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34647 } }
I20260809 00:53:56.474174 27352 heartbeater.cc:503] Master 127.9.225.62:35955 was elected leader, sending a full tablet report...
I20260809 00:53:56.474704 27582 heartbeater.cc:503] Master 127.9.225.62:35955 was elected leader, sending a full tablet report...
I20260809 00:53:56.475456 27613 ts_tablet_manager.cc:1428] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2: Time spent starting tablet: real 0.023s user 0.012s sys 0.012s
I20260809 00:53:56.476284 27611 ts_tablet_manager.cc:1428] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa: Time spent starting tablet: real 0.024s user 0.017s sys 0.000s
I20260809 00:53:56.478349 27610 ts_tablet_manager.cc:1428] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09: Time spent starting tablet: real 0.026s user 0.019s sys 0.004s
W20260809 00:53:56.547065 27583 tablet.cc:2369] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:53:56.549229 27130 tablet.cc:2369] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:53:56.624475 27622 raft_consensus.cc:482] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:53:56.624854 27622 raft_consensus.cc:504] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40149 } } peers { permanent_uuid: "66bac660662a4158918a9d574e889e09" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43829 } } peers { permanent_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34647 } }
I20260809 00:53:56.626890 27622 leader_election.cc:288] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66bac660662a4158918a9d574e889e09 (127.9.225.2:43829), e0cbd260f7424ab386ae3e6b75ae2aa2 (127.9.225.3:34647)
I20260809 00:53:56.634346 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.627329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:41715 (local address 127.9.225.2:43829)
0809 00:53:56.627621 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:56.627628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:56.628429 (+ 801us) server_negotiation.cc:407] Connection header received
0809 00:53:56.628642 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:56.628650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:56.628737 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:56.628808 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:56.629590 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.630330 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.632407 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.633187 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.633353 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.633501 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:56.633883 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 00:53:56.634043 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260809 00:53:56.634352 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.627458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:32769 (local address 127.9.225.3:34647)
0809 00:53:56.627706 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:56.627715 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:56.628529 (+ 814us) server_negotiation.cc:407] Connection header received
0809 00:53:56.628718 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:56.628722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:56.628818 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:56.628915 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:53:56.629575 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.630332 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.632370 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.633183 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.633347 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.633497 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:56.633875 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 00:53:56.634043 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 00:53:56.634677 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.627484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:34647 (local address 127.9.225.1:32769)
0809 00:53:56.628363 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:56.628405 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:56.628581 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:56.628970 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:56.628975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:56.629021 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:56.629400 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:56.629409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.630493 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.630498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:56.632175 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:56.632193 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.633544 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.633557 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:56.633685 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.633731 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 00:53:56.633842 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 00:53:56.634047 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":54,"thread_start_us":510,"threads_started":1}
I20260809 00:53:56.634832 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.627322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:43829 (local address 127.9.225.1:41715)
0809 00:53:56.628291 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0809 00:53:56.628307 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 00:53:56.628456 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:53:56.628861 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:53:56.628869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:53:56.628901 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:53:56.629416 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:56.629435 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.630474 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.630482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:56.632202 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:53:56.632215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.633499 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.633508 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:53:56.633672 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.633716 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 00:53:56.633840 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 00:53:56.634064 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":129,"thread_start_us":582,"threads_started":1}
I20260809 00:53:56.635787 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd84fa84710491fa93dd2074d11e799" candidate_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66bac660662a4158918a9d574e889e09" is_pre_election: true
I20260809 00:53:56.635787 27503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd84fa84710491fa93dd2074d11e799" candidate_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2" is_pre_election: true
I20260809 00:53:56.636348 27266 raft_consensus.cc:2454] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecbe7e05f1d49beb618ec597bf3c4fa in term 0.
I20260809 00:53:56.636351 27503 raft_consensus.cc:2454] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecbe7e05f1d49beb618ec597bf3c4fa in term 0.
I20260809 00:53:56.637091 26963 leader_election.cc:302] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecbe7e05f1d49beb618ec597bf3c4fa, e0cbd260f7424ab386ae3e6b75ae2aa2; no voters:
I20260809 00:53:56.637611 27622 raft_consensus.cc:2781] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:53:56.637773 27622 raft_consensus.cc:482] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:53:56.637894 27622 raft_consensus.cc:3037] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:56.640296 27622 raft_consensus.cc:504] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40149 } } peers { permanent_uuid: "66bac660662a4158918a9d574e889e09" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43829 } } peers { permanent_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34647 } }
I20260809 00:53:56.640967 27622 leader_election.cc:288] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [CANDIDATE]: Term 1 election: Requested vote from peers 66bac660662a4158918a9d574e889e09 (127.9.225.2:43829), e0cbd260f7424ab386ae3e6b75ae2aa2 (127.9.225.3:34647)
I20260809 00:53:56.641561 27266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd84fa84710491fa93dd2074d11e799" candidate_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66bac660662a4158918a9d574e889e09"
I20260809 00:53:56.641571 27503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bd84fa84710491fa93dd2074d11e799" candidate_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2"
I20260809 00:53:56.641798 27266 raft_consensus.cc:3037] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:56.641809 27503 raft_consensus.cc:3037] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:53:56.644196 27503 raft_consensus.cc:2454] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecbe7e05f1d49beb618ec597bf3c4fa in term 1.
I20260809 00:53:56.644394 27266 raft_consensus.cc:2454] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecbe7e05f1d49beb618ec597bf3c4fa in term 1.
I20260809 00:53:56.644724 26963 leader_election.cc:302] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecbe7e05f1d49beb618ec597bf3c4fa, e0cbd260f7424ab386ae3e6b75ae2aa2; no voters:
I20260809 00:53:56.645114 27622 raft_consensus.cc:2781] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:53:56.645681 27622 raft_consensus.cc:686] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 1 LEADER]: Becoming Leader. State: Replica: 4ecbe7e05f1d49beb618ec597bf3c4fa, State: Running, Role: LEADER
I20260809 00:53:56.646301 27622 consensus_queue.cc:237] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40149 } } peers { permanent_uuid: "66bac660662a4158918a9d574e889e09" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43829 } } peers { permanent_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34647 } }
I20260809 00:53:56.651558 26832 catalog_manager.cc:5495] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecbe7e05f1d49beb618ec597bf3c4fa (127.9.225.1). New cstate: current_term: 1 leader_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbe7e05f1d49beb618ec597bf3c4fa" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 40149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66bac660662a4158918a9d574e889e09" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34647 } health_report { overall_health: UNKNOWN } } }
W20260809 00:53:56.663821 27353 tablet.cc:2369] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:53:56.677008 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.670614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.9.225.1:40149)
0809 00:53:56.670852 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:56.670873 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:56.670895 (+ 22us) server_negotiation.cc:407] Connection header received
0809 00:53:56.671013 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:56.671021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:56.671126 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:56.671287 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:53:56.671661 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.672837 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.674402 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.674968 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.675077 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.675637 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:53:56.676226 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:53:56.676351 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:56.676553 (+ 202us) server_negotiation.cc:299] Negotiation successful
0809 00:53:56.676645 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 00:53:56.686808 27503 raft_consensus.cc:1261] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecbe7e05f1d49beb618ec597bf3c4fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:56.686808 27266 raft_consensus.cc:1261] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09 [term 1 FOLLOWER]: Refusing update from remote peer 4ecbe7e05f1d49beb618ec597bf3c4fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:53:56.688162 27626 consensus_queue.cc:1035] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cbd260f7424ab386ae3e6b75ae2aa2" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:56.688608 27622 consensus_queue.cc:1035] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "66bac660662a4158918a9d574e889e09" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:53:56.701397 27630 mvcc.cc:205] Tried to move back new op lower bound from 7316426083056574464 to 7316426082908065792. Current Snapshot: MvccSnapshot[applied={T|T < 7316426083056574464}]
I20260809 00:53:56.720697 26832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:56.721063 26832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:53:56.725670 26832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba378f1b372e41468c3053f63d5a1cdd: Sending DeleteTablet for 3 replicas of tablet 7bd84fa84710491fa93dd2074d11e799
I20260809 00:53:56.727185 27030 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bd84fa84710491fa93dd2074d11e799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:56 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:54622
I20260809 00:53:56.727176 27235 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bd84fa84710491fa93dd2074d11e799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:56 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:37544
I20260809 00:53:56.727234 27465 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bd84fa84710491fa93dd2074d11e799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:53:56 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:37492
I20260809 00:53:56.736063 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 26930
I20260809 00:53:56.740098 27642 tablet_replica.cc:330] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09: stopping tablet replica
I20260809 00:53:56.740716 27644 tablet_replica.cc:330] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2: stopping tablet replica
I20260809 00:53:56.740684 27642 raft_consensus.cc:2229] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:56.741325 27642 raft_consensus.cc:2258] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:56.741297 27644 raft_consensus.cc:2229] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:53:56.742079 27644 raft_consensus.cc:2258] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:56.743418 27643 tablet_replica.cc:330] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa: stopping tablet replica
I20260809 00:53:56.744028 27643 raft_consensus.cc:2229] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:53:56.745009 27643 raft_consensus.cc:2258] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:53:56.745119 27642 ts_tablet_manager.cc:1905] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:56.745119 27644 ts_tablet_manager.cc:1905] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:56.748035 27643 ts_tablet_manager.cc:1905] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:53:56.753489 27644 ts_tablet_manager.cc:1918] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:56.753830 27644 log.cc:1196] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/7bd84fa84710491fa93dd2074d11e799
I20260809 00:53:56.754515 27644 ts_tablet_manager.cc:1939] T 7bd84fa84710491fa93dd2074d11e799 P e0cbd260f7424ab386ae3e6b75ae2aa2: Deleting consensus metadata
I20260809 00:53:56.756924 26809 catalog_manager.cc:4841] TS e0cbd260f7424ab386ae3e6b75ae2aa2 (127.9.225.3:34647): tablet 7bd84fa84710491fa93dd2074d11e799 (table test-table [id=527831e5b539407e9ca53392ba42a470]) successfully deleted
I20260809 00:53:56.760129 27643 ts_tablet_manager.cc:1918] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:56.760475 27643 log.cc:1196] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/7bd84fa84710491fa93dd2074d11e799
I20260809 00:53:56.760812 27642 ts_tablet_manager.cc:1918] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:53:56.761003 27642 log.cc:1196] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/7bd84fa84710491fa93dd2074d11e799
I20260809 00:53:56.761418 27643 ts_tablet_manager.cc:1939] T 7bd84fa84710491fa93dd2074d11e799 P 4ecbe7e05f1d49beb618ec597bf3c4fa: Deleting consensus metadata
Aug 09 00:53:56 dist-test-slave-j2vl krb5kdc[26760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:53:56.761632 27642 ts_tablet_manager.cc:1939] T 7bd84fa84710491fa93dd2074d11e799 P 66bac660662a4158918a9d574e889e09: Deleting consensus metadata
I20260809 00:53:56.763737 26805 catalog_manager.cc:4841] TS 66bac660662a4158918a9d574e889e09 (127.9.225.2:43829): tablet 7bd84fa84710491fa93dd2074d11e799 (table test-table [id=527831e5b539407e9ca53392ba42a470]) successfully deleted
I20260809 00:53:56.763942 26808 catalog_manager.cc:4841] TS 4ecbe7e05f1d49beb618ec597bf3c4fa (127.9.225.1:40149): tablet 7bd84fa84710491fa93dd2074d11e799 (table test-table [id=527831e5b539407e9ca53392ba42a470]) successfully deleted
I20260809 00:53:56.770478 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.737864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.9.225.1:40149)
0809 00:53:56.738067 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:56.738073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:56.748644 (+ 10571us) server_negotiation.cc:407] Connection header received
0809 00:53:56.749224 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:56.749231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:56.749356 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:56.749452 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:56.751588 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.752947 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.755082 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.755835 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.755955 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.762961 (+ 7006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:56.762981 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:56.763005 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:56.763063 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:56.766957 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:56.768034 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:56.768043 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:56.768045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:56.768216 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:56.768686 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:56.768689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:56.768691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:56.769099 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:56.769420 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:56.769825 (+ 405us) server_negotiation.cc:299] Negotiation successful
0809 00:53:56.770079 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 00:53:56.771231 27120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:56.877511 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 26930
I20260809 00:53:56.903914 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 27137
Aug 09 00:53:56 dist-test-slave-j2vl krb5kdc[26760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:53:56.920512 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:56.904839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.9.225.2:43829)
0809 00:53:56.905038 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:56.905043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:56.905063 (+ 20us) server_negotiation.cc:407] Connection header received
0809 00:53:56.905121 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:56.905125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:56.905205 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:56.905291 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:56.906667 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.907525 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.908871 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:56.909297 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:56.909622 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:56.914144 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:56.914161 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:56.914181 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:56.914227 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:56.917111 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:56.918121 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:56.918130 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:56.918133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:56.918286 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:56.918713 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:56.918717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:56.918719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:56.919205 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:56.919436 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:56.919873 (+ 437us) server_negotiation.cc:299] Negotiation successful
0809 00:53:56.920176 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 00:53:56.921314 27342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:57.016638 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 27137
I20260809 00:53:57.046489 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 27364
Aug 09 00:53:57 dist-test-slave-j2vl krb5kdc[26760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:53:57.063717 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:53:57.047446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.9.225.3:34647)
0809 00:53:57.047664 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 00:53:57.047671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:53:57.047696 (+ 25us) server_negotiation.cc:407] Connection header received
0809 00:53:57.047878 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:53:57.047884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:53:57.047987 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:53:57.048117 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:53:57.049656 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:57.050428 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:57.051749 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:53:57.052224 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:53:57.052367 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:53:57.057389 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:53:57.057420 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:53:57.057440 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:53:57.057487 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:53:57.060434 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:57.061261 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:57.061279 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:57.061281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:57.061581 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:53:57.062010 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:53:57.062016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:53:57.062019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:53:57.062418 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:53:57.062749 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0809 00:53:57.063078 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 00:53:57.063326 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 00:53:57.064471 27576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:57.163142 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 27364
I20260809 00:53:57.193603 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 26778
I20260809 00:53:57.195614 26879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:53:57.319303 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 26778
2026-08-09T00:53:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5329 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:53:57 dist-test-slave-j2vl krb5kdc[27658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:53:57 dist-test-slave-j2vl krb5kdc[27658](info): set up 2 sockets
Aug 09 00:53:57 dist-test-slave-j2vl krb5kdc[27658](info): commencing operation
krb5kdc: starting...
W20260809 00:53:58.736986 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.366s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:53:58 dist-test-slave-j2vl krb5kdc[27658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:53:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.225.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.225.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/rangeradmin_127.9.225.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/rangeradmin_127.9.225.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.225.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.225.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/rangerlookup_127.9.225.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/rangerlookup_127.9.225.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.225.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.225.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/HTTP_127.9.225.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/HTTP_127.9.225.20@KRBTEST.COM.keytab.
I20260809 00:53:59.242350 10116 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskizkHom/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/postgres -l logfile start
2026-08-09 00:54:02.237 UTC [27701] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 00:54:02.237 UTC [27701] LOG: listening on IPv4 address "127.9.225.20", port 33855
2026-08-09 00:54:02.241 UTC [27701] LOG: listening on Unix socket "/tmp/.s.PGSQL.33855"
2026-08-09 00:54:02.269 UTC [27704] LOG: database system was shut down at 2026-08-09 00:54:00 UTC
2026-08-09 00:54:02.273 UTC [27701] LOG: database system is ready to accept connections
I20260809 00:54:03.493088 10116 mini_postgres.cc:96] Postgres bound to 33855
2026-08-09 00:54:03.521 UTC [27717] FATAL: database "slave" does not exist
127.9.225.20:33855 - accepting connections
I20260809 00:54:03.522691 10116 mini_ranger.cc:161] Starting Ranger...
I20260809 00:54:03.561156 10116 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 00:54:04.185444 10116 mini_ranger.cc:87] Created ranger Postgres database
I20260809 00:54:04.185611 10116 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-admin
2026-08-09 00:54:04,578 [I] DB FLAVOR :POSTGRES
2026-08-09 00:54:04,579 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 00:54:04,579 [I] Checking connection..
2026-08-09 00:54:04,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 00:54:04,873 [I] Checking connection passed.
2026-08-09 00:54:04,874 [I] --------- Verifying version history table ---------
2026-08-09 00:54:04,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 00:54:05,164 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 00:54:05,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 00:54:05,432 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 00:54:05,433 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 00:54:05,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 00:54:05.705 UTC [27817] WARNING: there is no transaction in progress
2026-08-09 00:54:05,713 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 00:54:05,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 00:54:06,009 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 00:54:06,010 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 00:54:06,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 00:54:06,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 00:54:06,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j2vl.c.gcp-upstream.internal','N') ;"
2026-08-09 00:54:06,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 00:54:07,129 [I] Table x_portal_user does not exist in database ranger
2026-08-09 00:54:07,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 00:54:07,417 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 00:54:07,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 00:54:07,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 00:54:07,939 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 00:54:07,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 00:54:09.161 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.183 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.204 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.222 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.243 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.264 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.329 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.337 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.355 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.367 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.379 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.394 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.406 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.414 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.424 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:09.433 UTC [27995] WARNING: there is no transaction in progress
2026-08-09 00:54:10,230 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 00:54:10,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 00:54:10,491 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 00:54:10,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 00:54:10,758 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 00:54:10,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 00:54:11,031 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 00:54:11,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 00:54:11,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 00:54:11,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j2vl.c.gcp-upstream.internal';"
2026-08-09 00:54:11,843 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 00:54:11,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 00:54:12,137 [I] DB_PATCHES have already been applied
I20260809 00:54:12.144306 10116 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-admin:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskizkHom/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 00:54:12.144457 10116 mini_ranger.cc:193] Using host: 127.9.225.20
I20260809 00:54:12.170099 10116 mini_ranger.cc:227] Ranger admin URL: http://127.9.225.20:49807
Aug 09, 2026 12:54:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 12:54:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 12:54:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 12:54:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 12:54:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 12:54:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 12:54:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 00:54:13.640967 10116 mini_ranger.cc:160] Time spent starting Ranger: real 10.118s user 0.008s sys 0.070s
Aug 09, 2026 12:54:35 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 12:54:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 09, 2026 12:54:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 12:54:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 12:54:35 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 12:54:36 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 00:54:37.111829 10116 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.225.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.225.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/rangerkms_127.9.225.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/rangerkms_127.9.225.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.225.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.225.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.225.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/HTTP_127.9.225.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.225.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/HTTP_127.9.225.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 00:54:37 dist-test-slave-j2vl krb5kdc[27658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236877, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 00:54:37.356612 10116 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 00:54:37.389899 10116 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 00:54:38.499518 10116 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 00:54:38.499711 10116 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-kms
I20260809 00:54:38.499859 10116 mini_ranger_kms.cc:227] Using postgres at 127.9.225.20:33855
2026-08-09 00:54:39,447 [I] DB FLAVOR :POSTGRES
2026-08-09 00:54:39,447 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 00:54:39,448 [I] Checking connection
2026-08-09 00:54:39,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 00:54:39,713 [I] connection success
2026-08-09 00:54:39,713 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 00:54:39,713 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 00:54:39,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 00:54:39,993 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 00:54:39,994 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 00:54:39,994 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 00:54:39,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:33855/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 00:54:40,290 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 00:54:40.880616 10116 mini_ranger_kms.cc:310] Created kms service
I20260809 00:54:41.079182 10116 mini_ranger_kms.cc:326] Created kudu user
I20260809 00:54:41.129835 10116 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 00:54:41 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236877, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:54:41.602475 10116 mini_ranger_kms.cc:384] Added ranger policy
I20260809 00:54:41.603984 10116 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-kms:/tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 00:54:41.604061 10116 mini_ranger_kms.cc:240] Using host: 127.9.225.20
I20260809 00:54:41.634095 10116 mini_ranger_kms.cc:276] Ranger KMS PID: 28280
I20260809 00:54:41.634291 10116 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.225.20:34305
Aug 09, 2026 12:54:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 12:54:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 12:54:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 12:54:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 12:54:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 12:54:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 00:54:43 dist-test-slave-j2vl krb5kdc[27658](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786236883, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.225.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 09 00:54:44 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236883, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.225.20@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:54:44.592029 10116 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 7.235s user 0.169s sys 0.526s
I20260809 00:54:44.592423 10116 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 00:54:44.592644 10116 mini_ranger_kms.cc:401] 127.9.225.20:34305/kms/v1/keys
Aug 09 00:54:44 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236883, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.225.20@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
Aug 09 00:54:47 dist-test-slave-j2vl krb5kdc[27658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:54:47.662345 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:44421
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:44421
--ranger_config_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.225.20:34305/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:54:48.227572 28355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:54:48.227923 28355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:54:48.228098 28355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:54:48.236064 28355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:54:48.236198 28355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:54:48.236302 28355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:54:48.236418 28355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:54:48.236500 28355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:54:48.248109 28355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.225.20:34305/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:44421
--ranger_config_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:44421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:54:48.249745 28355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:54:48.251295 28355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:54:48.265769 28369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:54:48.265704 28366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:54:48.266601 28355 server_base.cc:1034] running on GCE node
W20260809 00:54:48.266645 28364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:54:48.267807 28355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:54:48.270061 28355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:54:48.271374 28355 hybrid_clock.cc:648] HybridClock initialized: now 1786236888271311 us; error 58 us; skew 500 ppm
I20260809 00:54:48.271840 28355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:54:48 dist-test-slave-j2vl krb5kdc[27658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236888, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:54:48.281289 28355 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:54:48.283687 28355 webserver.cc:460] Webserver started at http://127.9.225.62:45549/ using document root <none> and password file <none>
I20260809 00:54:48.284839 28355 fs_manager.cc:359] Metadata directory not provided
I20260809 00:54:48.285127 28355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:54:48.285606 28355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 00:54:48 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236888, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:54:48.363929 28355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "0cb7da8accfd4e07b83838e8c2f5af98"
format_stamp: "Formatted at 2026-08-09 00:54:48 on dist-test-slave-j2vl"
server_key: "596407c3397c704ea5173ecba5a29814"
server_key_iv: "91547e55cbb043ff0fdce773b5627007"
server_key_version: "kuduclusterkey@0"
I20260809 00:54:48.365492 28355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "0cb7da8accfd4e07b83838e8c2f5af98"
format_stamp: "Formatted at 2026-08-09 00:54:48 on dist-test-slave-j2vl"
server_key: "596407c3397c704ea5173ecba5a29814"
server_key_iv: "91547e55cbb043ff0fdce773b5627007"
server_key_version: "kuduclusterkey@0"
I20260809 00:54:48.375141 28355 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260809 00:54:48.400797 28379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:54:48.402855 28355 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.003s
I20260809 00:54:48.403184 28355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "0cb7da8accfd4e07b83838e8c2f5af98"
format_stamp: "Formatted at 2026-08-09 00:54:48 on dist-test-slave-j2vl"
server_key: "596407c3397c704ea5173ecba5a29814"
server_key_iv: "91547e55cbb043ff0fdce773b5627007"
server_key_version: "kuduclusterkey@0"
I20260809 00:54:48.403486 28355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:54:48.433871 28355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:54:48.439189 28355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:54:48.439592 28355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:54:48.496699 28355 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:44421
I20260809 00:54:48.496912 28462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:44421 every 8 connection(s)
I20260809 00:54:48.498906 28355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:54:48.504995 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:54:48.505927 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 28355
I20260809 00:54:48.506156 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:54:48.507012 10116 external_mini_cluster.cc:1428] Setting key 734e2de913565a648f3d14e18f88b23e
I20260809 00:54:48.527647 28463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98: Bootstrap starting.
Aug 09 00:54:48 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:54:48.533161 28463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98: Neither blocks nor log segments found. Creating new log.
I20260809 00:54:48.536208 28463 log.cc:824] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98: Log is configured to *not* fsync() on all Append() calls
I20260809 00:54:48.542635 28463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98: No bootstrap required, opened a new log
I20260809 00:54:48.544327 28468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:48.510101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.9.225.62:44421)
0809 00:54:48.511933 (+ 1832us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:48.511958 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:48.512211 (+ 253us) server_negotiation.cc:407] Connection header received
0809 00:54:48.515289 (+ 3078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:48.515325 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:48.515948 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:48.516413 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:54:48.518916 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:48.520894 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:48.523320 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:48.524505 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:48.524682 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:48.530548 (+ 5866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:54:48.530561 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:54:48.530580 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:54:48.530664 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:54:48.536819 (+ 6155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:48.538436 (+ 1617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:48.538445 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:48.538457 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:48.538713 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:48.539348 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:48.539357 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:48.539364 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:48.540424 (+ 1060us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:54:48.541037 (+ 613us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:48.541604 (+ 567us) server_negotiation.cc:299] Negotiation successful
0809 00:54:48.542180 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211,"thread_start_us":811,"threads_started":1}
I20260809 00:54:48.556821 28463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb7da8accfd4e07b83838e8c2f5af98" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44421 } }
I20260809 00:54:48.557641 28463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:54:48.557780 28463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb7da8accfd4e07b83838e8c2f5af98, State: Initialized, Role: FOLLOWER
I20260809 00:54:48.558725 28463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb7da8accfd4e07b83838e8c2f5af98" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44421 } }
I20260809 00:54:48.559242 28463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:54:48.559576 28463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:54:48.559836 28463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:54:48.563588 28463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb7da8accfd4e07b83838e8c2f5af98" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44421 } }
I20260809 00:54:48.564567 28463 leader_election.cc:302] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cb7da8accfd4e07b83838e8c2f5af98; no voters:
I20260809 00:54:48.566004 28463 leader_election.cc:288] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:54:48.566273 28471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:54:48.568410 28471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [term 1 LEADER]: Becoming Leader. State: Replica: 0cb7da8accfd4e07b83838e8c2f5af98, State: Running, Role: LEADER
I20260809 00:54:48.569690 28463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:54:48.569476 28471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb7da8accfd4e07b83838e8c2f5af98" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44421 } }
I20260809 00:54:48.576596 28475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cb7da8accfd4e07b83838e8c2f5af98. Latest consensus state: current_term: 1 leader_uuid: "0cb7da8accfd4e07b83838e8c2f5af98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb7da8accfd4e07b83838e8c2f5af98" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44421 } } }
I20260809 00:54:48.576738 28472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cb7da8accfd4e07b83838e8c2f5af98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb7da8accfd4e07b83838e8c2f5af98" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44421 } } }
I20260809 00:54:48.578291 28472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [sys.catalog]: This master's current role is: LEADER
I20260809 00:54:48.578328 28475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98 [sys.catalog]: This master's current role is: LEADER
I20260809 00:54:48.584786 28483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:54:48.596465 28483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:54:48.599223 28463 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 00:54:48.638692 28483 catalog_manager.cc:1347] Generated new cluster ID: 78259f9e45a84fc3a2ca1938fd2b2904
I20260809 00:54:48.638954 28483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:54:48.665122 28483 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:54:48.668779 28483 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:54:48.686746 28483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98: Generated new TSK 0
I20260809 00:54:48.688778 28483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
W20260809 00:54:49.653047 10116 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.225.1: real 0.100s user 0.004s sys 0.031s
I20260809 00:54:49.654549 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.225.20:34305/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44421
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:54:49 dist-test-slave-j2vl krb5kdc[27658](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786236889, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 00:54:50.399915 28533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:54:50.400656 28533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:54:50.401000 28533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:54:50.415997 28533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:54:50.416224 28533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:54:50.416442 28533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:54:50.434420 28533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.225.20:34305/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:54:50.436762 28533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:54:50.439270 28533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:54:50.473830 28549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:54:50.479946 28553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:54:50.481238 28550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:54:50.486048 28533 server_base.cc:1034] running on GCE node
I20260809 00:54:50.487443 28533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:54:50.489466 28533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:54:50.493124 28533 hybrid_clock.cc:648] HybridClock initialized: now 1786236890492848 us; error 52 us; skew 500 ppm
I20260809 00:54:50.493937 28533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:54:50 dist-test-slave-j2vl krb5kdc[27658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:54:50.501832 28533 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:54:50.513243 28533 webserver.cc:460] Webserver started at http://127.9.225.1:41829/ using document root <none> and password file <none>
I20260809 00:54:50.514796 28533 fs_manager.cc:359] Metadata directory not provided
I20260809 00:54:50.515087 28533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:54:50.515687 28533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 00:54:50 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:54:50.549465 28533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "7f7f48f5b6d2449f8a16e499aa123549"
format_stamp: "Formatted at 2026-08-09 00:54:50 on dist-test-slave-j2vl"
server_key: "05a2079ddabd551257bc41db161a60dd"
server_key_iv: "bf2385b383429dc0a768f2485d2fb68e"
server_key_version: "kuduclusterkey@0"
I20260809 00:54:50.551368 28533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "7f7f48f5b6d2449f8a16e499aa123549"
format_stamp: "Formatted at 2026-08-09 00:54:50 on dist-test-slave-j2vl"
server_key: "05a2079ddabd551257bc41db161a60dd"
server_key_iv: "bf2385b383429dc0a768f2485d2fb68e"
server_key_version: "kuduclusterkey@0"
I20260809 00:54:50.572957 28533 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.016s sys 0.004s
I20260809 00:54:50.606266 28562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:54:50.608938 28533 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.006s
I20260809 00:54:50.609421 28533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "7f7f48f5b6d2449f8a16e499aa123549"
format_stamp: "Formatted at 2026-08-09 00:54:50 on dist-test-slave-j2vl"
server_key: "05a2079ddabd551257bc41db161a60dd"
server_key_iv: "bf2385b383429dc0a768f2485d2fb68e"
server_key_version: "kuduclusterkey@0"
I20260809 00:54:50.609972 28533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:54:50.664714 28533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:54:50.673841 28533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:54:50.674795 28533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:54:50.683698 28533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:54:50.692898 28533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:54:50.693140 28533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 00:54:50.693384 28533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:54:50.693521 28533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 00:54:50 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236889, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:54:50.915115 28533 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:42491
I20260809 00:54:50.915191 28746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:42491 every 8 connection(s)
I20260809 00:54:50.917164 28533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:54:50.924113 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 28533
I20260809 00:54:50.924353 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:54:50.925827 10116 external_mini_cluster.cc:1428] Setting key 2f882db7f0977f387d966bf13c304af7
Aug 09 00:54:50 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236890, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:54:50.969125 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:50.924449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44421 (local address 127.9.225.1:38401)
0809 00:54:50.926330 (+ 1881us) negotiation.cc:107] Waiting for socket to connect
0809 00:54:50.926414 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0809 00:54:50.931338 (+ 4924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:54:50.935569 (+ 4231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:54:50.935596 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:54:50.936867 (+ 1271us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:54:50.938462 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:50.938516 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:50.944899 (+ 6383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:50.944907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:50.948310 (+ 3403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:50.948344 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:50.950344 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:50.950356 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:50.950766 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:50.952212 (+ 1446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:54:50.952263 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:54:50.956490 (+ 4227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:54:50.962656 (+ 6166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:54:50.962669 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:54:50.962692 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:54:50.963439 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:54:50.963938 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:54:50.963943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:54:50.963946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:54:50.964416 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:54:50.965214 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:54:50.965223 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:54:50.965936 (+ 713us) client_negotiation.cc:769] Sending connection context
0809 00:54:50.966383 (+ 447us) client_negotiation.cc:240] Negotiation successful
0809 00:54:50.967194 (+ 811us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1441,"thread_start_us":1152,"threads_started":1}
I20260809 00:54:50.971706 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:50.925464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:38401 (local address 127.9.225.62:44421)
0809 00:54:50.934431 (+ 8967us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:50.934438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:50.934479 (+ 41us) server_negotiation.cc:407] Connection header received
0809 00:54:50.934585 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:50.934591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:50.934744 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:50.938832 (+ 4088us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:54:50.939242 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:50.940714 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:50.948688 (+ 7974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:50.949675 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:50.949880 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:50.956918 (+ 7038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:54:50.956927 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:54:50.956932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:54:50.956984 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:54:50.962384 (+ 5400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:50.963627 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:50.963634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:50.963637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:50.963801 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:50.964582 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:50.964586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:50.964589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:50.965055 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:54:50.970507 (+ 5452us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:50.970696 (+ 189us) server_negotiation.cc:299] Negotiation successful
0809 00:54:50.971055 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8740,"thread_start_us":8386,"threads_started":1}
I20260809 00:54:50.974365 28747 heartbeater.cc:347] Connected to a master server at 127.9.225.62:44421
I20260809 00:54:50.975072 28747 heartbeater.cc:464] Registering TS with master...
I20260809 00:54:50.976814 28747 heartbeater.cc:511] Master 127.9.225.62:44421 requested a full tablet report, sending...
I20260809 00:54:50.981076 28403 ts_manager.cc:194] Registered new tserver with Master: 7f7f48f5b6d2449f8a16e499aa123549 (127.9.225.1:42491)
I20260809 00:54:50.984062 28403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:38401
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
W20260809 00:54:51.041556 10116 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.225.2: real 0.105s user 0.001s sys 0.037s
I20260809 00:54:51.042757 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.225.20:34305/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44421
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:54:51 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236889, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
W20260809 00:54:51.786237 28756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:54:51.786873 28756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:54:51.787202 28756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:54:51.804195 28756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:54:51.804503 28756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:54:51.804936 28756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:54:51.826342 28756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.225.20:34305/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:54:51.828781 28756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:54:51.831130 28756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:54:51.860140 28782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:54:51.860458 28784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:54:51.861748 28786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:54:51.862705 28756 server_base.cc:1034] running on GCE node
I20260809 00:54:51.863899 28756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:54:51.865517 28756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:54:51.866838 28756 hybrid_clock.cc:648] HybridClock initialized: now 1786236891866724 us; error 77 us; skew 500 ppm
I20260809 00:54:51.867458 28756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:54:51 dist-test-slave-j2vl krb5kdc[27658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:54:51.874127 28756 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:54:51.876953 28756 webserver.cc:460] Webserver started at http://127.9.225.2:43299/ using document root <none> and password file <none>
I20260809 00:54:51.878713 28756 fs_manager.cc:359] Metadata directory not provided
I20260809 00:54:51.879109 28756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:54:51.879876 28756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 00:54:51 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:54:51.903504 28756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "d47d118e8354489e8371fbfd7fa2bc3c"
format_stamp: "Formatted at 2026-08-09 00:54:51 on dist-test-slave-j2vl"
server_key: "4c9c5232d249e3425a59dd16bab393d0"
server_key_iv: "e4e4c229286878c6a2c651bc17a8619e"
server_key_version: "kuduclusterkey@0"
I20260809 00:54:51.905125 28756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "d47d118e8354489e8371fbfd7fa2bc3c"
format_stamp: "Formatted at 2026-08-09 00:54:51 on dist-test-slave-j2vl"
server_key: "4c9c5232d249e3425a59dd16bab393d0"
server_key_iv: "e4e4c229286878c6a2c651bc17a8619e"
server_key_version: "kuduclusterkey@0"
I20260809 00:54:51.914925 28756 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.002s
I20260809 00:54:51.937702 28800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:54:51.940032 28756 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260809 00:54:51.940362 28756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "d47d118e8354489e8371fbfd7fa2bc3c"
format_stamp: "Formatted at 2026-08-09 00:54:51 on dist-test-slave-j2vl"
server_key: "4c9c5232d249e3425a59dd16bab393d0"
server_key_iv: "e4e4c229286878c6a2c651bc17a8619e"
server_key_version: "kuduclusterkey@0"
I20260809 00:54:51.940744 28756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:54:51.976365 28756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:54:51.983568 28756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:54:51.984030 28756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:54:51.985857 28756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:54:51.988245 28756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:54:51.988435 28756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:54:51.988670 28756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:54:51.988842 28756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:54:51.989475 28747 heartbeater.cc:503] Master 127.9.225.62:44421 was elected leader, sending a full tablet report...
I20260809 00:54:52.081691 28756 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:43873
I20260809 00:54:52.081856 28975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:43873 every 8 connection(s)
I20260809 00:54:52.083436 28756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:54:52.084829 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 28756
I20260809 00:54:52.085071 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:54:52.085861 10116 external_mini_cluster.cc:1428] Setting key 66b67818f863c9687073f73c9099b9fa
Aug 09 00:54:52 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236891, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:54:52.120790 28979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:52.087754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:41233 (local address 127.9.225.62:44421)
0809 00:54:52.088869 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:52.088876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:52.093310 (+ 4434us) server_negotiation.cc:407] Connection header received
0809 00:54:52.095935 (+ 2625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:52.095945 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:52.096086 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:52.096476 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:54:52.098780 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:52.100798 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:52.103399 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:52.104403 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:52.104589 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:52.110422 (+ 5833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:54:52.110432 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:54:52.110439 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:54:52.110743 (+ 304us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:54:52.115552 (+ 4809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:52.117177 (+ 1625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:52.117186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:52.117190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:52.117559 (+ 369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:52.118324 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:52.118329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:52.118331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:52.118730 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:54:52.118906 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:52.120108 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0809 00:54:52.120330 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":655,"threads_started":1}
I20260809 00:54:52.122857 28980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:52.090184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44421 (local address 127.9.225.2:41233)
0809 00:54:52.092165 (+ 1981us) negotiation.cc:107] Waiting for socket to connect
0809 00:54:52.092229 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 00:54:52.095468 (+ 3239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:54:52.096738 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:54:52.096749 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:54:52.097565 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:54:52.098535 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:52.098562 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:52.101046 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:52.101051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:52.103179 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:52.103199 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:52.105025 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:52.105115 (+ 90us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:52.105625 (+ 510us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:52.106600 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:54:52.106630 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:54:52.109987 (+ 3357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:54:52.115826 (+ 5839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:54:52.115844 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:54:52.115871 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:54:52.116791 (+ 920us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:54:52.117720 (+ 929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:54:52.117729 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:54:52.117736 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:54:52.118148 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:54:52.118912 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:54:52.118927 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:54:52.119545 (+ 618us) client_negotiation.cc:769] Sending connection context
0809 00:54:52.119933 (+ 388us) client_negotiation.cc:240] Negotiation successful
0809 00:54:52.120721 (+ 788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1501,"thread_start_us":1012,"threads_started":1}
I20260809 00:54:52.125222 28976 heartbeater.cc:347] Connected to a master server at 127.9.225.62:44421
I20260809 00:54:52.125742 28976 heartbeater.cc:464] Registering TS with master...
I20260809 00:54:52.127187 28976 heartbeater.cc:511] Master 127.9.225.62:44421 requested a full tablet report, sending...
I20260809 00:54:52.129791 28403 ts_manager.cc:194] Registered new tserver with Master: d47d118e8354489e8371fbfd7fa2bc3c (127.9.225.2:43873)
I20260809 00:54:52.130815 28403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:41233
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:54:52.182462 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.225.20:34305/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44421
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:54:52.846778 28984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:54:52.847240 28984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:54:52.847507 28984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:54:52.857860 28984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:54:52.858115 28984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:54:52.858366 28984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:54:52.872231 28984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.225.20:34305/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:54:52.874172 28984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:54:52.876051 28984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:54:52.900667 28993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:54:52.901970 28990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:54:52.902086 28991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:54:52.902956 28984 server_base.cc:1034] running on GCE node
I20260809 00:54:52.904281 28984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:54:52.906324 28984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:54:52.907618 28984 hybrid_clock.cc:648] HybridClock initialized: now 1786236892907482 us; error 99 us; skew 500 ppm
I20260809 00:54:52.908227 28984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:54:52 dist-test-slave-j2vl krb5kdc[27658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236892, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:54:52.915992 28984 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:54:52.923316 28984 webserver.cc:460] Webserver started at http://127.9.225.3:44357/ using document root <none> and password file <none>
I20260809 00:54:52.924388 28984 fs_manager.cc:359] Metadata directory not provided
I20260809 00:54:52.924641 28984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:54:52.925136 28984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 00:54:52 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236892, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:54:52.947280 28984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "ade369b6895e4b409059d0aee101a7aa"
format_stamp: "Formatted at 2026-08-09 00:54:52 on dist-test-slave-j2vl"
server_key: "c8744a1727c5d14f653e37c296bedee8"
server_key_iv: "e01444d08ad13e04cba683567d8b3d2c"
server_key_version: "kuduclusterkey@0"
I20260809 00:54:52.949081 28984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "ade369b6895e4b409059d0aee101a7aa"
format_stamp: "Formatted at 2026-08-09 00:54:52 on dist-test-slave-j2vl"
server_key: "c8744a1727c5d14f653e37c296bedee8"
server_key_iv: "e01444d08ad13e04cba683567d8b3d2c"
server_key_version: "kuduclusterkey@0"
I20260809 00:54:52.959029 28984 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.002s
I20260809 00:54:52.977541 29001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:54:52.979554 28984 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 00:54:52.979946 28984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "ade369b6895e4b409059d0aee101a7aa"
format_stamp: "Formatted at 2026-08-09 00:54:52 on dist-test-slave-j2vl"
server_key: "c8744a1727c5d14f653e37c296bedee8"
server_key_iv: "e01444d08ad13e04cba683567d8b3d2c"
server_key_version: "kuduclusterkey@0"
I20260809 00:54:52.980314 28984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:54:53.007546 28984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:54:53.013823 28984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:54:53.014430 28984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:54:53.015952 28984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:54:53.018788 28984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:54:53.018962 28984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:54:53.019217 28984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:54:53.019375 28984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:54:53.121518 28984 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:37037
I20260809 00:54:53.121536 29172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:37037 every 8 connection(s)
I20260809 00:54:53.123163 28984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:54:53.128661 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 28984
I20260809 00:54:53.128876 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:54:53.129501 10116 external_mini_cluster.cc:1428] Setting key e25e603d0deffb654f141de8bc94f4c2
I20260809 00:54:53.135000 28976 heartbeater.cc:503] Master 127.9.225.62:44421 was elected leader, sending a full tablet report...
Aug 09 00:54:53 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236892, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:54:53.156131 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.126994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:48663 (local address 127.9.225.62:44421)
0809 00:54:53.128215 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:53.128224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:53.129676 (+ 1452us) server_negotiation.cc:407] Connection header received
0809 00:54:53.134593 (+ 4917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:53.134599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:53.134704 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:53.135087 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:54:53.138430 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.139631 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.141775 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.142283 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.142401 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.148200 (+ 5799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:54:53.148207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:54:53.148210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:54:53.148309 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:54:53.152118 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:53.153148 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:53.153153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:53.153156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:53.153466 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:53.154024 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:53.154028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:53.154030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:53.154473 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:54:53.154631 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:53.155532 (+ 901us) server_negotiation.cc:299] Negotiation successful
0809 00:54:53.155730 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1055,"thread_start_us":816,"threads_started":1}
I20260809 00:54:53.157717 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.127474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44421 (local address 127.9.225.3:48663)
0809 00:54:53.129180 (+ 1706us) negotiation.cc:107] Waiting for socket to connect
0809 00:54:53.129299 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0809 00:54:53.134093 (+ 4794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:54:53.135538 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:54:53.135568 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:54:53.136705 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:54:53.138220 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:53.138249 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.139794 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.139800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:53.141567 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:53.141616 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.142710 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.142722 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:53.143068 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.144093 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:54:53.144153 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:54:53.147939 (+ 3786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:54:53.152309 (+ 4370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:54:53.152319 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:54:53.152337 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:54:53.152979 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:54:53.153607 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:54:53.153611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:54:53.153614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:54:53.153891 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:54:53.154605 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:54:53.154614 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:54:53.155029 (+ 415us) client_negotiation.cc:769] Sending connection context
0809 00:54:53.155383 (+ 354us) client_negotiation.cc:240] Negotiation successful
0809 00:54:53.155930 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1328,"thread_start_us":943,"threads_started":1}
I20260809 00:54:53.160077 29173 heartbeater.cc:347] Connected to a master server at 127.9.225.62:44421
I20260809 00:54:53.160532 29173 heartbeater.cc:464] Registering TS with master...
I20260809 00:54:53.161743 29173 heartbeater.cc:511] Master 127.9.225.62:44421 requested a full tablet report, sending...
I20260809 00:54:53.164124 28403 ts_manager.cc:194] Registered new tserver with Master: ade369b6895e4b409059d0aee101a7aa (127.9.225.3:37037)
I20260809 00:54:53.165323 28403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:48663
I20260809 00:54:53.173518 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:54:53.203164 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.180853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.9.225.62:44421)
0809 00:54:53.181046 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:53.181051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:53.182301 (+ 1250us) server_negotiation.cc:407] Connection header received
0809 00:54:53.182557 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:53.182563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:53.182664 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:53.182782 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:54:53.185572 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.186572 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.188306 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.189103 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.189268 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.194197 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:54:53.194206 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:54:53.194209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:54:53.194259 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:54:53.198493 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:53.199863 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:53.199869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:53.199872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:53.200109 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:53.200707 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:53.200712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:53.200715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:53.201184 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:54:53.201616 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:53.202103 (+ 487us) server_negotiation.cc:299] Negotiation successful
0809 00:54:53.202418 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 00:54:53.209803 28403 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:54:53.221962 28403 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:54:53.278707 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.244351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.9.225.2:43873)
0809 00:54:53.257936 (+ 13585us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:53.257952 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:53.260846 (+ 2894us) server_negotiation.cc:407] Connection header received
0809 00:54:53.262151 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:53.262165 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:53.262601 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:53.262945 (+ 344us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:54:53.265051 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.268046 (+ 2995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.273954 (+ 5908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.276381 (+ 2427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.276616 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.276996 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:53.277713 (+ 717us) server_negotiation.cc:299] Negotiation successful
0809 00:54:53.278024 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12781,"thread_start_us":13030,"threads_started":1}
I20260809 00:54:53.280213 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.243253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:43873 (local address 127.0.0.1:48898)
0809 00:54:53.260660 (+ 17407us) negotiation.cc:107] Waiting for socket to connect
0809 00:54:53.260683 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:54:53.261061 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:54:53.263746 (+ 2685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:54:53.263755 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:54:53.263799 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:54:53.264698 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:53.264716 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.268332 (+ 3616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.268339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:53.273705 (+ 5366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:53.273723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.276997 (+ 3274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.277005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:53.277378 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.277442 (+ 64us) client_negotiation.cc:769] Sending connection context
0809 00:54:53.279008 (+ 1566us) client_negotiation.cc:240] Negotiation successful
0809 00:54:53.279276 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17136,"mutex_wait_us":79,"thread_start_us":17919,"threads_started":1}
I20260809 00:54:53.281620 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.242816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:37037 (local address 127.0.0.1:58246)
0809 00:54:53.259638 (+ 16822us) negotiation.cc:107] Waiting for socket to connect
0809 00:54:53.259716 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0809 00:54:53.260072 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:54:53.262198 (+ 2126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:54:53.262206 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:54:53.262237 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:54:53.263454 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:53.263477 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.267284 (+ 3807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.267298 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:53.272663 (+ 5365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:53.272680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.276804 (+ 4124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.276817 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:53.277231 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.280364 (+ 3133us) client_negotiation.cc:769] Sending connection context
0809 00:54:53.280465 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 00:54:53.280696 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16364,"mutex_wait_us":24,"thread_start_us":15809,"threads_started":1}
I20260809 00:54:53.281759 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.243628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.9.225.3:37037)
0809 00:54:53.259744 (+ 16116us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:53.259754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:53.259854 (+ 100us) server_negotiation.cc:407] Connection header received
0809 00:54:53.260544 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:53.260564 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:53.260964 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:53.261311 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:54:53.263895 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.266692 (+ 2797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.272899 (+ 6207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.275761 (+ 2862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.275957 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.276453 (+ 496us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:53.280647 (+ 4194us) server_negotiation.cc:299] Negotiation successful
0809 00:54:53.280928 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15834,"thread_start_us":15519,"threads_started":1}
I20260809 00:54:53.284675 29197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.260186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:42491 (local address 127.0.0.1:47316)
0809 00:54:53.268142 (+ 7956us) negotiation.cc:107] Waiting for socket to connect
0809 00:54:53.268171 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 00:54:53.268329 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:54:53.271031 (+ 2702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:54:53.271038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:54:53.271089 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:54:53.272924 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:53.272951 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.276045 (+ 3094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.276051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:53.279670 (+ 3619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:53.279688 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.283106 (+ 3418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.283115 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:53.283395 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.283435 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 00:54:53.283543 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 00:54:53.283697 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7727,"mutex_wait_us":2,"thread_start_us":7276,"threads_started":1}
I20260809 00:54:53.285441 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.260823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.9.225.1:42491)
0809 00:54:53.269970 (+ 9147us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:53.269982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:53.270019 (+ 37us) server_negotiation.cc:407] Connection header received
0809 00:54:53.270194 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:53.270216 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:53.270662 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:53.271094 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:54:53.273186 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.275827 (+ 2641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.279981 (+ 4154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.282184 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.282321 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.282701 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:53.284536 (+ 1835us) server_negotiation.cc:299] Negotiation successful
0809 00:54:53.284793 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8772,"thread_start_us":8896,"threads_started":1}
I20260809 00:54:53.287308 29078 tablet_service.cc:1467] Processing CreateTablet for tablet b7075c1e8c7c41edb60f743649aff7a1 (DEFAULT_TABLE table=test-table [id=0b0ccd6cb705485f94e8f93d91329479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:54:53.287278 28870 tablet_service.cc:1467] Processing CreateTablet for tablet b7075c1e8c7c41edb60f743649aff7a1 (DEFAULT_TABLE table=test-table [id=0b0ccd6cb705485f94e8f93d91329479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:54:53.289893 29078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7075c1e8c7c41edb60f743649aff7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:54:53.290160 28870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7075c1e8c7c41edb60f743649aff7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:54:53.291417 28640 tablet_service.cc:1467] Processing CreateTablet for tablet b7075c1e8c7c41edb60f743649aff7a1 (DEFAULT_TABLE table=test-table [id=0b0ccd6cb705485f94e8f93d91329479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:54:53.293965 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7075c1e8c7c41edb60f743649aff7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:54:53.310164 29200 tablet_bootstrap.cc:492] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa: Bootstrap starting.
I20260809 00:54:53.322283 29200 tablet_bootstrap.cc:654] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa: Neither blocks nor log segments found. Creating new log.
I20260809 00:54:53.326118 29200 log.cc:824] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa: Log is configured to *not* fsync() on all Append() calls
I20260809 00:54:53.326932 29202 tablet_bootstrap.cc:492] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549: Bootstrap starting.
I20260809 00:54:53.327550 29201 tablet_bootstrap.cc:492] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c: Bootstrap starting.
I20260809 00:54:53.335232 29202 tablet_bootstrap.cc:654] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549: Neither blocks nor log segments found. Creating new log.
I20260809 00:54:53.336278 29201 tablet_bootstrap.cc:654] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c: Neither blocks nor log segments found. Creating new log.
I20260809 00:54:53.337491 29200 tablet_bootstrap.cc:492] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa: No bootstrap required, opened a new log
I20260809 00:54:53.338122 29200 ts_tablet_manager.cc:1397] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.000s
I20260809 00:54:53.338454 29202 log.cc:824] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549: Log is configured to *not* fsync() on all Append() calls
I20260809 00:54:53.338555 29201 log.cc:824] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c: Log is configured to *not* fsync() on all Append() calls
I20260809 00:54:53.344154 29201 tablet_bootstrap.cc:492] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c: No bootstrap required, opened a new log
I20260809 00:54:53.344641 29201 ts_tablet_manager.cc:1397] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.009s
I20260809 00:54:53.345904 29202 tablet_bootstrap.cc:492] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549: No bootstrap required, opened a new log
I20260809 00:54:53.346364 29202 ts_tablet_manager.cc:1397] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.009s
I20260809 00:54:53.355580 29200 raft_consensus.cc:348] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade369b6895e4b409059d0aee101a7aa" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37037 } } peers { permanent_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43873 } } peers { permanent_uuid: "7f7f48f5b6d2449f8a16e499aa123549" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42491 } }
I20260809 00:54:53.356300 29200 raft_consensus.cc:374] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:54:53.356503 29200 raft_consensus.cc:729] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade369b6895e4b409059d0aee101a7aa, State: Initialized, Role: FOLLOWER
I20260809 00:54:53.357851 29200 consensus_queue.cc:260] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade369b6895e4b409059d0aee101a7aa" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37037 } } peers { permanent_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43873 } } peers { permanent_uuid: "7f7f48f5b6d2449f8a16e499aa123549" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42491 } }
I20260809 00:54:53.359751 29202 raft_consensus.cc:348] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade369b6895e4b409059d0aee101a7aa" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37037 } } peers { permanent_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43873 } } peers { permanent_uuid: "7f7f48f5b6d2449f8a16e499aa123549" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42491 } }
I20260809 00:54:53.360287 29202 raft_consensus.cc:374] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:54:53.360411 29173 heartbeater.cc:503] Master 127.9.225.62:44421 was elected leader, sending a full tablet report...
I20260809 00:54:53.360437 29202 raft_consensus.cc:729] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7f48f5b6d2449f8a16e499aa123549, State: Initialized, Role: FOLLOWER
I20260809 00:54:53.361277 29200 ts_tablet_manager.cc:1428] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa: Time spent starting tablet: real 0.023s user 0.013s sys 0.009s
I20260809 00:54:53.361532 29202 consensus_queue.cc:260] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade369b6895e4b409059d0aee101a7aa" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37037 } } peers { permanent_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43873 } } peers { permanent_uuid: "7f7f48f5b6d2449f8a16e499aa123549" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42491 } }
I20260809 00:54:53.363932 29201 raft_consensus.cc:348] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade369b6895e4b409059d0aee101a7aa" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37037 } } peers { permanent_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43873 } } peers { permanent_uuid: "7f7f48f5b6d2449f8a16e499aa123549" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42491 } }
I20260809 00:54:53.364493 29201 raft_consensus.cc:374] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:54:53.364622 29201 raft_consensus.cc:729] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47d118e8354489e8371fbfd7fa2bc3c, State: Initialized, Role: FOLLOWER
I20260809 00:54:53.365533 29201 consensus_queue.cc:260] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade369b6895e4b409059d0aee101a7aa" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37037 } } peers { permanent_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43873 } } peers { permanent_uuid: "7f7f48f5b6d2449f8a16e499aa123549" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42491 } }
I20260809 00:54:53.370330 29202 ts_tablet_manager.cc:1428] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549: Time spent starting tablet: real 0.024s user 0.021s sys 0.003s
I20260809 00:54:53.373776 29201 ts_tablet_manager.cc:1428] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c: Time spent starting tablet: real 0.029s user 0.021s sys 0.007s
W20260809 00:54:53.376129 29175 tablet.cc:2369] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:54:53.423377 28748 tablet.cc:2369] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:54:53.483139 29210 raft_consensus.cc:482] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:54:53.483559 29210 raft_consensus.cc:504] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade369b6895e4b409059d0aee101a7aa" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37037 } } peers { permanent_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43873 } } peers { permanent_uuid: "7f7f48f5b6d2449f8a16e499aa123549" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42491 } }
I20260809 00:54:53.486066 29210 leader_election.cc:288] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ade369b6895e4b409059d0aee101a7aa (127.9.225.3:37037), 7f7f48f5b6d2449f8a16e499aa123549 (127.9.225.1:42491)
I20260809 00:54:53.497795 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.486603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:48191 (local address 127.9.225.3:37037)
0809 00:54:53.486886 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:53.486897 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:53.487695 (+ 798us) server_negotiation.cc:407] Connection header received
0809 00:54:53.488120 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:53.488140 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:53.488259 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:53.488348 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:54:53.490066 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.491111 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.494386 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.496098 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.496354 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.496567 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:53.497276 (+ 709us) server_negotiation.cc:299] Negotiation successful
0809 00:54:53.497451 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 00:54:53.498324 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.486287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:37037 (local address 127.9.225.2:48191)
0809 00:54:53.487574 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0809 00:54:53.487592 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:54:53.487866 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:54:53.488716 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:54:53.488723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:54:53.488812 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:54:53.489834 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:53.489872 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.491283 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.491290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:53.494182 (+ 2892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:53.494200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.496521 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.496530 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:53.497010 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.497085 (+ 75us) client_negotiation.cc:769] Sending connection context
0809 00:54:53.497228 (+ 143us) client_negotiation.cc:240] Negotiation successful
0809 00:54:53.497452 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":30,"thread_start_us":614,"threads_started":1}
I20260809 00:54:53.498770 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.487815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:59981 (local address 127.9.225.1:42491)
0809 00:54:53.488112 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:53.488121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:53.488827 (+ 706us) server_negotiation.cc:407] Connection header received
0809 00:54:53.489073 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:53.489078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:53.489167 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:53.489247 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:54:53.490831 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.491739 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.494852 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.496146 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.496330 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.496520 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:53.497585 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0809 00:54:53.497814 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260809 00:54:53.499075 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.487439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:42491 (local address 127.9.225.2:59981)
0809 00:54:53.488708 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0809 00:54:53.488726 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:54:53.488922 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:54:53.489304 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:54:53.489311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:54:53.489380 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:54:53.490580 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:53.490592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.491957 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.491961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:53.494611 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:54:53.494626 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.497024 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.497032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:54:53.497388 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.497421 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 00:54:53.498136 (+ 715us) client_negotiation.cc:240] Negotiation successful
0809 00:54:53.498300 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":6,"thread_start_us":686,"threads_started":1}
I20260809 00:54:53.500994 29108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7075c1e8c7c41edb60f743649aff7a1" candidate_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade369b6895e4b409059d0aee101a7aa" is_pre_election: true
I20260809 00:54:53.500959 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7075c1e8c7c41edb60f743649aff7a1" candidate_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f7f48f5b6d2449f8a16e499aa123549" is_pre_election: true
I20260809 00:54:53.502020 29108 raft_consensus.cc:2454] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d47d118e8354489e8371fbfd7fa2bc3c in term 0.
I20260809 00:54:53.501988 28673 raft_consensus.cc:2454] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d47d118e8354489e8371fbfd7fa2bc3c in term 0.
I20260809 00:54:53.502998 28801 leader_election.cc:302] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ade369b6895e4b409059d0aee101a7aa, d47d118e8354489e8371fbfd7fa2bc3c; no voters:
I20260809 00:54:53.503651 29210 raft_consensus.cc:2781] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:54:53.503980 29210 raft_consensus.cc:482] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:54:53.504160 29210 raft_consensus.cc:3037] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:54:53.509603 29210 raft_consensus.cc:504] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade369b6895e4b409059d0aee101a7aa" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37037 } } peers { permanent_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43873 } } peers { permanent_uuid: "7f7f48f5b6d2449f8a16e499aa123549" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42491 } }
I20260809 00:54:53.510842 29210 leader_election.cc:288] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [CANDIDATE]: Term 1 election: Requested vote from peers ade369b6895e4b409059d0aee101a7aa (127.9.225.3:37037), 7f7f48f5b6d2449f8a16e499aa123549 (127.9.225.1:42491)
I20260809 00:54:53.511588 29108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7075c1e8c7c41edb60f743649aff7a1" candidate_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade369b6895e4b409059d0aee101a7aa"
I20260809 00:54:53.511742 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7075c1e8c7c41edb60f743649aff7a1" candidate_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f7f48f5b6d2449f8a16e499aa123549"
I20260809 00:54:53.512050 28673 raft_consensus.cc:3037] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:54:53.512066 29108 raft_consensus.cc:3037] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:54:53.518127 28673 raft_consensus.cc:2454] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d47d118e8354489e8371fbfd7fa2bc3c in term 1.
I20260809 00:54:53.518147 29108 raft_consensus.cc:2454] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d47d118e8354489e8371fbfd7fa2bc3c in term 1.
I20260809 00:54:53.518911 28801 leader_election.cc:302] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f7f48f5b6d2449f8a16e499aa123549, d47d118e8354489e8371fbfd7fa2bc3c; no voters:
I20260809 00:54:53.519480 29210 raft_consensus.cc:2781] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:54:53.520522 29210 raft_consensus.cc:686] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 1 LEADER]: Becoming Leader. State: Replica: d47d118e8354489e8371fbfd7fa2bc3c, State: Running, Role: LEADER
I20260809 00:54:53.521492 29210 consensus_queue.cc:237] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade369b6895e4b409059d0aee101a7aa" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37037 } } peers { permanent_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43873 } } peers { permanent_uuid: "7f7f48f5b6d2449f8a16e499aa123549" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42491 } }
I20260809 00:54:53.528925 28403 catalog_manager.cc:5495] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c reported cstate change: term changed from 0 to 1, leader changed from <none> to d47d118e8354489e8371fbfd7fa2bc3c (127.9.225.2). New cstate: current_term: 1 leader_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ade369b6895e4b409059d0aee101a7aa" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d47d118e8354489e8371fbfd7fa2bc3c" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f7f48f5b6d2449f8a16e499aa123549" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42491 } health_report { overall_health: UNKNOWN } } }
I20260809 00:54:53.584596 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.576635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.9.225.2:43873)
0809 00:54:53.576925 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:53.576930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:53.576950 (+ 20us) server_negotiation.cc:407] Connection header received
0809 00:54:53.577061 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:53.577064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:53.577160 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:53.577316 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:54:53.578071 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.579159 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.581456 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.582098 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.582217 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.583072 (+ 855us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:54:53.583833 (+ 761us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:54:53.583956 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:53.584207 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 00:54:53.584303 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260809 00:54:53.595618 28978 tablet.cc:2369] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:54:53.598948 29108 raft_consensus.cc:1261] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa [term 1 FOLLOWER]: Refusing update from remote peer d47d118e8354489e8371fbfd7fa2bc3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:54:53.598922 28673 raft_consensus.cc:1261] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549 [term 1 FOLLOWER]: Refusing update from remote peer d47d118e8354489e8371fbfd7fa2bc3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:54:53.600368 29214 consensus_queue.cc:1035] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade369b6895e4b409059d0aee101a7aa" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:54:53.601094 29210 consensus_queue.cc:1035] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f7f48f5b6d2449f8a16e499aa123549" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 42491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:54:53.650992 28403 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:54:53.651433 28403 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:54:53.657003 28403 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cb7da8accfd4e07b83838e8c2f5af98: Sending DeleteTablet for 3 replicas of tablet b7075c1e8c7c41edb60f743649aff7a1
I20260809 00:54:53.659148 29078 tablet_service.cc:1514] Processing DeleteTablet for tablet b7075c1e8c7c41edb60f743649aff7a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:54:53 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:58246
I20260809 00:54:53.659144 28870 tablet_service.cc:1514] Processing DeleteTablet for tablet b7075c1e8c7c41edb60f743649aff7a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:54:53 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:48898
I20260809 00:54:53.660393 28640 tablet_service.cc:1514] Processing DeleteTablet for tablet b7075c1e8c7c41edb60f743649aff7a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:54:53 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:47316
I20260809 00:54:53.661021 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 28533
I20260809 00:54:53.661468 29232 tablet_replica.cc:330] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa: stopping tablet replica
I20260809 00:54:53.662451 29232 raft_consensus.cc:2229] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:54:53.663744 29232 raft_consensus.cc:2258] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:54:53.669584 29232 ts_tablet_manager.cc:1905] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:54:53.673456 29234 tablet_replica.cc:330] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549: stopping tablet replica
I20260809 00:54:53.674118 29234 raft_consensus.cc:2229] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:54:53.674773 29234 raft_consensus.cc:2258] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:54:53.677275 29234 ts_tablet_manager.cc:1905] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:54:53.682107 29233 tablet_replica.cc:330] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c: stopping tablet replica
I20260809 00:54:53.682708 29233 raft_consensus.cc:2229] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:54:53.685626 29234 ts_tablet_manager.cc:1918] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:54:53.685928 29234 log.cc:1196] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/b7075c1e8c7c41edb60f743649aff7a1
I20260809 00:54:53.686233 29232 ts_tablet_manager.cc:1918] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:54:53.686550 29234 ts_tablet_manager.cc:1939] T b7075c1e8c7c41edb60f743649aff7a1 P 7f7f48f5b6d2449f8a16e499aa123549: Deleting consensus metadata
I20260809 00:54:53.686594 29232 log.cc:1196] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/b7075c1e8c7c41edb60f743649aff7a1
I20260809 00:54:53.687889 29232 ts_tablet_manager.cc:1939] T b7075c1e8c7c41edb60f743649aff7a1 P ade369b6895e4b409059d0aee101a7aa: Deleting consensus metadata
I20260809 00:54:53.689536 28382 catalog_manager.cc:4841] TS 7f7f48f5b6d2449f8a16e499aa123549 (127.9.225.1:42491): tablet b7075c1e8c7c41edb60f743649aff7a1 (table test-table [id=0b0ccd6cb705485f94e8f93d91329479]) successfully deleted
I20260809 00:54:53.690577 28382 catalog_manager.cc:4841] TS ade369b6895e4b409059d0aee101a7aa (127.9.225.3:37037): tablet b7075c1e8c7c41edb60f743649aff7a1 (table test-table [id=0b0ccd6cb705485f94e8f93d91329479]) successfully deleted
I20260809 00:54:53.691128 29233 raft_consensus.cc:2258] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:54:53.693885 29233 ts_tablet_manager.cc:1905] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 00:54:53 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:54:53.703716 29233 ts_tablet_manager.cc:1918] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:54:53.704105 29233 log.cc:1196] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/b7075c1e8c7c41edb60f743649aff7a1
I20260809 00:54:53.705132 29233 ts_tablet_manager.cc:1939] T b7075c1e8c7c41edb60f743649aff7a1 P d47d118e8354489e8371fbfd7fa2bc3c: Deleting consensus metadata
I20260809 00:54:53.707839 28386 catalog_manager.cc:4841] TS d47d118e8354489e8371fbfd7fa2bc3c (127.9.225.2:43873): tablet b7075c1e8c7c41edb60f743649aff7a1 (table test-table [id=0b0ccd6cb705485f94e8f93d91329479]) successfully deleted
I20260809 00:54:53.712527 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.662152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.9.225.1:42491)
0809 00:54:53.662352 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:53.662361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:53.673053 (+ 10692us) server_negotiation.cc:407] Connection header received
0809 00:54:53.678897 (+ 5844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:53.678903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:53.679007 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:53.679110 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:54:53.683302 (+ 4192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.684649 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.687845 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.688791 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.688906 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.701234 (+ 12328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:54:53.701280 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:54:53.701322 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:54:53.701507 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:54:53.707671 (+ 6164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:53.709150 (+ 1479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:53.709161 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:53.709190 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:53.709546 (+ 356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:53.710249 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:53.710253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:53.710255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:53.710884 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:54:53.711244 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:53.711855 (+ 611us) server_negotiation.cc:299] Negotiation successful
0809 00:54:53.712065 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 00:54:53.713863 28738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:54:53.812589 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 28533
I20260809 00:54:53.855296 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 28756
Aug 09 00:54:53 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:54:53.874579 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:53.856167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.9.225.2:43873)
0809 00:54:53.856362 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:53.856368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:53.856479 (+ 111us) server_negotiation.cc:407] Connection header received
0809 00:54:53.856701 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:53.856706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:53.856788 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:53.856941 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:54:53.858816 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.859672 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.861302 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:53.861941 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:53.862111 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:53.866834 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:54:53.866862 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:54:53.866883 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:54:53.866935 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:54:53.870674 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:53.871632 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:53.871644 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:53.871651 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:53.871844 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:53.872274 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:53.872280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:53.872284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:53.872727 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:54:53.873208 (+ 481us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:53.873556 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 00:54:53.873912 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 00:54:53.875412 28968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:54:53.995437 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 28756
I20260809 00:54:54.031550 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 28984
Aug 09 00:54:54 dist-test-slave-j2vl krb5kdc[27658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:54:54.049863 29240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:54:54.032402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.9.225.3:37037)
0809 00:54:54.032960 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0809 00:54:54.032965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:54:54.032981 (+ 16us) server_negotiation.cc:407] Connection header received
0809 00:54:54.033038 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:54:54.033042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:54:54.033103 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:54:54.033227 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:54:54.034738 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:54.035493 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:54.036924 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:54:54.037470 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:54:54.037619 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:54:54.043194 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:54:54.043214 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:54:54.043235 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:54:54.043296 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:54:54.046552 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:54.047403 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:54.047411 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:54.047414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:54.047568 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:54:54.047994 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:54:54.047999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:54:54.048004 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:54:54.048470 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:54:54.048760 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0809 00:54:54.049158 (+ 398us) server_negotiation.cc:299] Negotiation successful
0809 00:54:54.049464 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":259,"threads_started":1}
I20260809 00:54:54.050627 29166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:54:54.145246 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 28984
I20260809 00:54:54.174257 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 28355
I20260809 00:54:54.175602 28452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:54:54.296844 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 28355
I20260809 00:54:54.335907 10116 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 00:54:54.682832 10116 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 00:54:54.683220 10116 mini_ranger.cc:66] Stopping Ranger...
I20260809 00:54:55.030258 10116 mini_ranger.cc:68] Stopped Ranger
2026-08-09 00:54:55.030 UTC [27701] LOG: received smart shutdown request
2026-08-09 00:54:55.035 UTC [27701] LOG: background worker "logical replication launcher" (PID 27714) exited with exit code 1
2026-08-09 00:54:55.035 UTC [27708] LOG: shutting down
2026-08-09 00:54:55.253 UTC [27701] LOG: database system is shut down
2026-08-09T00:54:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (58067 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:54:55 dist-test-slave-j2vl krb5kdc[29254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:54:55 dist-test-slave-j2vl krb5kdc[29254](info): set up 2 sockets
Aug 09 00:54:55 dist-test-slave-j2vl krb5kdc[29254](info): commencing operation
krb5kdc: starting...
W20260809 00:54:56.802999 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.366s user 0.004s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:54:57 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:54:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.225.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.225.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/rangeradmin_127.9.225.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/rangeradmin_127.9.225.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.225.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.225.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/rangerlookup_127.9.225.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/rangerlookup_127.9.225.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.225.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.225.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/HTTP_127.9.225.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/HTTP_127.9.225.20@KRBTEST.COM.keytab.
I20260809 00:54:57.264688 10116 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskizkHom/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/postgres -l logfile start
2026-08-09 00:55:00.305 UTC [29299] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 00:55:00.305 UTC [29299] LOG: listening on IPv4 address "127.9.225.20", port 52391
2026-08-09 00:55:00.310 UTC [29299] LOG: listening on Unix socket "/tmp/.s.PGSQL.52391"
2026-08-09 00:55:00.338 UTC [29302] LOG: database system was shut down at 2026-08-09 00:54:58 UTC
2026-08-09 00:55:00.342 UTC [29299] LOG: database system is ready to accept connections
I20260809 00:55:01.594228 10116 mini_postgres.cc:96] Postgres bound to 52391
2026-08-09 00:55:01.620 UTC [29326] FATAL: database "slave" does not exist
127.9.225.20:52391 - accepting connections
I20260809 00:55:01.621759 10116 mini_ranger.cc:161] Starting Ranger...
I20260809 00:55:01.658795 10116 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 00:55:02.291486 10116 mini_ranger.cc:87] Created ranger Postgres database
I20260809 00:55:02.291626 10116 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-admin
2026-08-09 00:55:02,610 [I] DB FLAVOR :POSTGRES
2026-08-09 00:55:02,610 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 00:55:02,610 [I] Checking connection..
2026-08-09 00:55:02,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 00:55:02,904 [I] Checking connection passed.
2026-08-09 00:55:02,904 [I] --------- Verifying version history table ---------
2026-08-09 00:55:02,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 00:55:03,180 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 00:55:03,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 00:55:03,473 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 00:55:03,474 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 00:55:03,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 00:55:03.751 UTC [29429] WARNING: there is no transaction in progress
2026-08-09 00:55:03,763 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 00:55:03,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 00:55:04,045 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 00:55:04,046 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 00:55:04,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 00:55:04,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 00:55:04,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j2vl.c.gcp-upstream.internal','N') ;"
2026-08-09 00:55:04,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 00:55:05,197 [I] Table x_portal_user does not exist in database ranger
2026-08-09 00:55:05,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 00:55:05,466 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 00:55:05,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 00:55:05,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 00:55:06,025 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 00:55:06,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 00:55:07.188 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.209 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.230 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.251 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.275 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.296 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.364 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.374 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.391 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.405 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.419 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.435 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.447 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.455 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.464 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:07.472 UTC [29606] WARNING: there is no transaction in progress
2026-08-09 00:55:08,316 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 00:55:08,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 00:55:08,577 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 00:55:08,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 00:55:08,898 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 00:55:08,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 00:55:09,170 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 00:55:09,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 00:55:09,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 00:55:09,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j2vl.c.gcp-upstream.internal';"
2026-08-09 00:55:10,004 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 00:55:10,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 00:55:10,272 [I] DB_PATCHES have already been applied
I20260809 00:55:10.279394 10116 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-admin:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskizkHom/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 00:55:10.279506 10116 mini_ranger.cc:193] Using host: 127.9.225.20
I20260809 00:55:10.302595 10116 mini_ranger.cc:227] Ranger admin URL: http://127.9.225.20:53323
Aug 09, 2026 12:55:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 12:55:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 12:55:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 12:55:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 12:55:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 12:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 12:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 00:55:11.804620 10116 mini_ranger.cc:160] Time spent starting Ranger: real 10.183s user 0.002s sys 0.069s
Aug 09, 2026 12:55:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 12:55:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 09, 2026 12:55:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 12:55:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 12:55:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 12:55:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 00:55:36.192284 10116 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.225.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.225.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/rangerkms_127.9.225.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.225.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/rangerkms_127.9.225.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.225.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.225.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.225.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/HTTP_127.9.225.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.225.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/HTTP_127.9.225.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 00:55:36 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236936, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 00:55:36.428190 10116 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 00:55:36.459532 10116 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 00:55:37.477591 10116 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 00:55:37.477736 10116 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-kms
I20260809 00:55:37.477782 10116 mini_ranger_kms.cc:227] Using postgres at 127.9.225.20:52391
2026-08-09 00:55:37,926 [I] DB FLAVOR :POSTGRES
2026-08-09 00:55:37,926 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 00:55:37,926 [I] Checking connection
2026-08-09 00:55:37,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 00:55:38,254 [I] connection success
2026-08-09 00:55:38,254 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 00:55:38,254 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 00:55:38,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 00:55:38,593 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 00:55:38,594 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 00:55:38,594 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 00:55:38,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.225.20:52391/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 00:55:38,906 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 00:55:39.463780 10116 mini_ranger_kms.cc:310] Created kms service
I20260809 00:55:39.726161 10116 mini_ranger_kms.cc:326] Created kudu user
I20260809 00:55:39.779640 10116 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 00:55:39 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236936, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:55:40.241109 10116 mini_ranger_kms.cc:384] Added ranger policy
I20260809 00:55:40.241509 10116 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-kms:/tmp/dist-test-taskizkHom/build/asan/bin/postgresql.jar:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskizkHom/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 00:55:40.241580 10116 mini_ranger_kms.cc:240] Using host: 127.9.225.20
I20260809 00:55:40.264860 10116 mini_ranger_kms.cc:276] Ranger KMS PID: 29909
I20260809 00:55:40.265013 10116 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.225.20:33065
Aug 09, 2026 12:55:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 12:55:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 12:55:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 12:55:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 12:55:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 12:55:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 00:55:42 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786236942, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.225.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:43.189795 10116 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.762s user 0.091s sys 0.383s
I20260809 00:55:43.190074 10116 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 00:55:43.190249 10116 mini_ranger_kms.cc:401] 127.9.225.20:33065/kms/v1/keys
Aug 09 00:55:43 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236942, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.225.20@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
Aug 09 00:55:43 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236942, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.225.20@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
Aug 09 00:55:46 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:55:46.375546 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35523
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:35523
--ranger_config_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.225.20:33065/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:55:46.770947 29992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:55:46.771327 29992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:55:46.771515 29992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:55:46.779709 29992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:55:46.779898 29992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:55:46.780028 29992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:55:46.780131 29992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:55:46.780222 29992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:55:46.792798 29992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.225.20:33065/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:35523
--ranger_config_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:55:46.794479 29992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:55:46.796073 29992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:55:46.810060 29997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:46.810150 30002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:46.810127 30000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:46.810860 29992 server_base.cc:1034] running on GCE node
I20260809 00:55:46.811861 29992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:55:46.813735 29992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:55:46.814992 29992 hybrid_clock.cc:648] HybridClock initialized: now 1786236946814953 us; error 45 us; skew 500 ppm
I20260809 00:55:46.815446 29992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:55:46 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236946, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:46.819434 29992 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:55:46.821557 29992 webserver.cc:460] Webserver started at http://127.9.225.62:40301/ using document root <none> and password file <none>
I20260809 00:55:46.822363 29992 fs_manager.cc:359] Metadata directory not provided
I20260809 00:55:46.822518 29992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:55:46.822885 29992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 00:55:46 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236946, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:55:46.884784 29992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "8e31e32134c74e288f0398c883eefad2"
format_stamp: "Formatted at 2026-08-09 00:55:46 on dist-test-slave-j2vl"
server_key: "8850178fb033bf0f961090e8ba15ffe2"
server_key_iv: "637a88c8238615a2a86c421368a07b83"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:46.885845 29992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "8e31e32134c74e288f0398c883eefad2"
format_stamp: "Formatted at 2026-08-09 00:55:46 on dist-test-slave-j2vl"
server_key: "8850178fb033bf0f961090e8ba15ffe2"
server_key_iv: "637a88c8238615a2a86c421368a07b83"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:46.892549 29992 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260809 00:55:46.911073 30011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:46.912803 29992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:55:46.913008 29992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "8e31e32134c74e288f0398c883eefad2"
format_stamp: "Formatted at 2026-08-09 00:55:46 on dist-test-slave-j2vl"
server_key: "8850178fb033bf0f961090e8ba15ffe2"
server_key_iv: "637a88c8238615a2a86c421368a07b83"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:46.913280 29992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:55:46.936952 29992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:55:46.941699 29992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:55:46.942070 29992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:55:46.965943 29992 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:35523
I20260809 00:55:46.965981 30092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:35523 every 8 connection(s)
I20260809 00:55:46.967556 29992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:55:46.971926 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 29992
I20260809 00:55:46.972219 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:55:46.973001 10116 external_mini_cluster.cc:1428] Setting key a27a3da59a199525bc3abac2903fd5c8
I20260809 00:55:46.974078 30094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:55:46.992360 30094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2: Bootstrap starting.
I20260809 00:55:46.997068 30094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2: Neither blocks nor log segments found. Creating new log.
I20260809 00:55:46.998546 30094 log.cc:824] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2: Log is configured to *not* fsync() on all Append() calls
Aug 09 00:55:46 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:55:47.002678 30094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2: No bootstrap required, opened a new log
I20260809 00:55:47.010242 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:46.975671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.9.225.62:35523)
0809 00:55:46.988010 (+ 12339us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:46.988027 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:46.988084 (+ 57us) server_negotiation.cc:407] Connection header received
0809 00:55:46.990413 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:46.990450 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:46.991000 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:46.991539 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:46.993235 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:46.994749 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:46.996224 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:46.997004 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:46.997412 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:47.002038 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:47.002047 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:47.002065 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:47.002105 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:47.005334 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:47.006218 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:47.006228 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:47.006237 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:47.006412 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:47.006858 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:47.006862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:47.006864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:47.007470 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:47.007853 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:47.008360 (+ 507us) server_negotiation.cc:299] Negotiation successful
0809 00:55:47.008945 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12000,"thread_start_us":11665,"threads_started":1}
I20260809 00:55:47.015725 30094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e31e32134c74e288f0398c883eefad2" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35523 } }
I20260809 00:55:47.016319 30094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:55:47.016458 30094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e31e32134c74e288f0398c883eefad2, State: Initialized, Role: FOLLOWER
I20260809 00:55:47.017251 30094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e31e32134c74e288f0398c883eefad2" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35523 } }
I20260809 00:55:47.017607 30094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:55:47.017767 30094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:55:47.018013 30094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:55:47.021766 30094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e31e32134c74e288f0398c883eefad2" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35523 } }
I20260809 00:55:47.022563 30094 leader_election.cc:302] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e31e32134c74e288f0398c883eefad2; no voters:
I20260809 00:55:47.023306 30094 leader_election.cc:288] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:55:47.023427 30102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:55:47.024809 30102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 1 LEADER]: Becoming Leader. State: Replica: 8e31e32134c74e288f0398c883eefad2, State: Running, Role: LEADER
I20260809 00:55:47.025537 30102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e31e32134c74e288f0398c883eefad2" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35523 } }
I20260809 00:55:47.026032 30094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:55:47.028849 30105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e31e32134c74e288f0398c883eefad2. Latest consensus state: current_term: 1 leader_uuid: "8e31e32134c74e288f0398c883eefad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e31e32134c74e288f0398c883eefad2" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35523 } } }
I20260809 00:55:47.029253 30105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [sys.catalog]: This master's current role is: LEADER
I20260809 00:55:47.028863 30104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e31e32134c74e288f0398c883eefad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e31e32134c74e288f0398c883eefad2" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35523 } } }
I20260809 00:55:47.029826 30104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [sys.catalog]: This master's current role is: LEADER
I20260809 00:55:47.043146 30108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:55:47.049945 30108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:55:47.064311 30108 catalog_manager.cc:1347] Generated new cluster ID: 4725c7067dd5453188e30df5bd88fc4c
I20260809 00:55:47.064471 30108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:55:47.086834 30094 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 00:55:47.164062 30108 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:55:47.166391 30108 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:55:47.177321 30108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2: Generated new TSK 0
I20260809 00:55:47.178622 30108 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:55:48.002718 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.225.20:33065/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35523
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:55:48 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786236948, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 00:55:48.454909 30158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:55:48.455420 30158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:55:48.455612 30158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:55:48.468116 30158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:55:48.468367 30158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:55:48.468605 30158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:55:48.487110 30158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.225.20:33065/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:55:48.489123 30158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:55:48.491287 30158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:55:48.511183 30176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:48.514386 30174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:48.517943 30158 server_base.cc:1034] running on GCE node
W20260809 00:55:48.518579 30178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:48.519536 30158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:55:48.521404 30158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:55:48.522991 30158 hybrid_clock.cc:648] HybridClock initialized: now 1786236948522906 us; error 76 us; skew 500 ppm
I20260809 00:55:48.523463 30158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:55:48 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236948, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:48.531672 30158 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:55:48.555876 30158 webserver.cc:460] Webserver started at http://127.9.225.1:33003/ using document root <none> and password file <none>
I20260809 00:55:48.557212 30158 fs_manager.cc:359] Metadata directory not provided
I20260809 00:55:48.557507 30158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:55:48.558012 30158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 00:55:48 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236948, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:55:48.589577 30158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "5ee8eec859b940dead68aac2c0f5f8f4"
format_stamp: "Formatted at 2026-08-09 00:55:48 on dist-test-slave-j2vl"
server_key: "655c5ef0f7a3e879376c9541907143ef"
server_key_iv: "cace6fd2e3c7b507f75c1c6b47b5a9fc"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:48.591044 30158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "5ee8eec859b940dead68aac2c0f5f8f4"
format_stamp: "Formatted at 2026-08-09 00:55:48 on dist-test-slave-j2vl"
server_key: "655c5ef0f7a3e879376c9541907143ef"
server_key_iv: "cace6fd2e3c7b507f75c1c6b47b5a9fc"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:48.600241 30158 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260809 00:55:48.626400 30187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:48.639014 30158 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.002s
I20260809 00:55:48.639467 30158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "5ee8eec859b940dead68aac2c0f5f8f4"
format_stamp: "Formatted at 2026-08-09 00:55:48 on dist-test-slave-j2vl"
server_key: "655c5ef0f7a3e879376c9541907143ef"
server_key_iv: "cace6fd2e3c7b507f75c1c6b47b5a9fc"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:48.639935 30158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:55:48.688135 30158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:55:48.694587 30158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:55:48.695124 30158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:55:48.696630 30158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:55:48.698906 30158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:55:48.699088 30158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 00:55:48.699316 30158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:55:48.699460 30158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:48.784749 30158 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:33633
I20260809 00:55:48.784760 30397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:33633 every 8 connection(s)
I20260809 00:55:48.786477 30158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:55:48.793453 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 30158
I20260809 00:55:48.793699 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:55:48.794575 10116 external_mini_cluster.cc:1428] Setting key 4f7674dadd89c2531d46bf6bba5b69c5
Aug 09 00:55:48 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236948, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:55:48.821112 30402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:48.790909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35523 (local address 127.9.225.1:51833)
0809 00:55:48.791907 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:48.791961 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:48.794705 (+ 2744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:48.796097 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:48.796107 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:48.796636 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:55:48.797321 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:48.797392 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:48.799016 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:48.799022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:48.802059 (+ 3037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:48.802089 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:48.803767 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:48.803778 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:48.804199 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:48.805191 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:55:48.805233 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:55:48.808660 (+ 3427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:55:48.814712 (+ 6052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:48.814745 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:48.814772 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:48.815626 (+ 854us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:48.816226 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:48.816235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:48.816241 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:48.816570 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:48.817618 (+ 1048us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:55:48.817628 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:55:48.818311 (+ 683us) client_negotiation.cc:769] Sending connection context
0809 00:55:48.818698 (+ 387us) client_negotiation.cc:240] Negotiation successful
0809 00:55:48.819408 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":421,"threads_started":1}
I20260809 00:55:48.822355 30400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:48.790315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:51833 (local address 127.9.225.62:35523)
0809 00:55:48.790852 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:48.790860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:48.792105 (+ 1245us) server_negotiation.cc:407] Connection header received
0809 00:55:48.795184 (+ 3079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:48.795192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:48.795364 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:48.795478 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:48.797596 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:48.798758 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:48.802337 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:48.803157 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:48.803293 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:48.809062 (+ 5769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:48.809074 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:48.809081 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:48.809149 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:48.814122 (+ 4973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:48.815837 (+ 1715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:48.815846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:48.815852 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:48.816060 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:48.816831 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:48.816837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:48.816840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:48.817273 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:48.817540 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:48.821634 (+ 4094us) server_negotiation.cc:299] Negotiation successful
0809 00:55:48.821954 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":259,"threads_started":1}
I20260809 00:55:48.824958 30398 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35523
I20260809 00:55:48.825368 30398 heartbeater.cc:464] Registering TS with master...
I20260809 00:55:48.826355 30398 heartbeater.cc:511] Master 127.9.225.62:35523 requested a full tablet report, sending...
I20260809 00:55:48.829322 30039 ts_manager.cc:194] Registered new tserver with Master: 5ee8eec859b940dead68aac2c0f5f8f4 (127.9.225.1:33633)
I20260809 00:55:48.831622 30039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:51833
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Aug 09 00:55:48 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236948, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
W20260809 00:55:48.908420 10116 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.225.2: real 0.108s user 0.000s sys 0.028s
I20260809 00:55:48.909502 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.225.20:33065/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35523
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:55:49 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236948, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
W20260809 00:55:49.472147 30408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:55:49.472743 30408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:55:49.472975 30408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:55:49.485432 30408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:55:49.485713 30408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:55:49.485998 30408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:55:49.503119 30408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.225.20:33065/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:55:49.505599 30408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:55:49.507611 30408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:55:49.526667 30418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:49.527628 30419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:49.533787 30423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:49.533923 30408 server_base.cc:1034] running on GCE node
I20260809 00:55:49.535006 30408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:55:49.536705 30408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:55:49.538061 30408 hybrid_clock.cc:648] HybridClock initialized: now 1786236949537995 us; error 54 us; skew 500 ppm
I20260809 00:55:49.538542 30408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:55:49 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236949, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:49.543493 30408 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:55:49.547335 30408 webserver.cc:460] Webserver started at http://127.9.225.2:43719/ using document root <none> and password file <none>
I20260809 00:55:49.548609 30408 fs_manager.cc:359] Metadata directory not provided
I20260809 00:55:49.548872 30408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:55:49.549533 30408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 00:55:49 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236949, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:55:49.571144 30408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "efab3b7a9a9942efafb119e28faa0642"
format_stamp: "Formatted at 2026-08-09 00:55:49 on dist-test-slave-j2vl"
server_key: "b157a175502b757408d9949e81dba97c"
server_key_iv: "c4b74676485f430dc40740f0ea38a36e"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:49.572634 30408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "efab3b7a9a9942efafb119e28faa0642"
format_stamp: "Formatted at 2026-08-09 00:55:49 on dist-test-slave-j2vl"
server_key: "b157a175502b757408d9949e81dba97c"
server_key_iv: "c4b74676485f430dc40740f0ea38a36e"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:49.584702 30408 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.002s
I20260809 00:55:49.614895 30444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:49.617385 30408 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.009s sys 0.004s
I20260809 00:55:49.617700 30408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "efab3b7a9a9942efafb119e28faa0642"
format_stamp: "Formatted at 2026-08-09 00:55:49 on dist-test-slave-j2vl"
server_key: "b157a175502b757408d9949e81dba97c"
server_key_iv: "c4b74676485f430dc40740f0ea38a36e"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:49.618063 30408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:55:49.647291 30408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:55:49.652861 30408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:55:49.653239 30408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:55:49.654600 30408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:55:49.656843 30408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:55:49.657001 30408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:49.657173 30408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:55:49.657258 30408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:49.697160 30408 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:40699
I20260809 00:55:49.697173 30624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:40699 every 8 connection(s)
I20260809 00:55:49.699229 30408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:55:49.709654 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 30408
I20260809 00:55:49.709861 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:55:49.710415 10116 external_mini_cluster.cc:1428] Setting key 9b7d8b5f7a015f5e22f3beb4abf18356
Aug 09 00:55:49 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236949, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:55:49.727437 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:49.702701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:34983 (local address 127.9.225.62:35523)
0809 00:55:49.703248 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:49.703254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:49.704704 (+ 1450us) server_negotiation.cc:407] Connection header received
0809 00:55:49.707039 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:49.707046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:49.707161 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:49.707302 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:49.708976 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:49.710195 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:49.712361 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:49.712994 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:49.713134 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:49.718607 (+ 5473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:49.718618 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:49.718626 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:49.718677 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:49.722934 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:49.724112 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:49.724120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:49.724126 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:49.724326 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:49.725146 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:49.725155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:49.725161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:49.725669 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:49.725905 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:49.726800 (+ 895us) server_negotiation.cc:299] Negotiation successful
0809 00:55:49.727130 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":270,"threads_started":1}
I20260809 00:55:49.728899 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:49.703486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35523 (local address 127.9.225.2:34983)
0809 00:55:49.704476 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:49.704543 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:49.706694 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:49.707561 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:49.707570 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:49.708081 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:55:49.708755 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:49.708775 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:49.710423 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:49.710430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:49.712196 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:49.712211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:49.713308 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:49.713318 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:49.713798 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:49.714966 (+ 1168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:55:49.715000 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:55:49.718195 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:55:49.723166 (+ 4971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:49.723178 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:49.723202 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:49.723934 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:49.724580 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:49.724586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:49.724590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:49.724945 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:49.725850 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:55:49.725860 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:55:49.726347 (+ 487us) client_negotiation.cc:769] Sending connection context
0809 00:55:49.726695 (+ 348us) client_negotiation.cc:240] Negotiation successful
0809 00:55:49.727394 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":434,"threads_started":1}
I20260809 00:55:49.730719 30627 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35523
I20260809 00:55:49.731124 30627 heartbeater.cc:464] Registering TS with master...
I20260809 00:55:49.732136 30627 heartbeater.cc:511] Master 127.9.225.62:35523 requested a full tablet report, sending...
I20260809 00:55:49.734243 30039 ts_manager.cc:194] Registered new tserver with Master: efab3b7a9a9942efafb119e28faa0642 (127.9.225.2:40699)
I20260809 00:55:49.735452 30039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:34983
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:55:49.789651 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.225.20:33065/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35523
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:55:49.836341 30398 heartbeater.cc:503] Master 127.9.225.62:35523 was elected leader, sending a full tablet report...
W20260809 00:55:50.283222 30634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:55:50.283728 30634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:55:50.283995 30634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:55:50.294679 30634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:55:50.294876 30634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:55:50.295080 30634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:55:50.309006 30634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.225.20:33065/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:55:50.310850 30634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:55:50.312500 30634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:55:50.331569 30643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:50.331506 30642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:50.331598 30645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:50.333443 30634 server_base.cc:1034] running on GCE node
I20260809 00:55:50.334156 30634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:55:50.335260 30634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:55:50.336494 30634 hybrid_clock.cc:648] HybridClock initialized: now 1786236950336447 us; error 32 us; skew 500 ppm
I20260809 00:55:50.336934 30634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:55:50 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:50.344789 30634 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:55:50.346896 30634 webserver.cc:460] Webserver started at http://127.9.225.3:41333/ using document root <none> and password file <none>
I20260809 00:55:50.347831 30634 fs_manager.cc:359] Metadata directory not provided
I20260809 00:55:50.347962 30634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:55:50.348340 30634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 00:55:50 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:55:50.368520 30634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "1214beb866ba4414857f3476898ca05f"
format_stamp: "Formatted at 2026-08-09 00:55:50 on dist-test-slave-j2vl"
server_key: "a265abbb03687c17dc2780763677f18b"
server_key_iv: "27aa15c0c2b0dcdf76e98ffabfeba0cc"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:50.370534 30634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "1214beb866ba4414857f3476898ca05f"
format_stamp: "Formatted at 2026-08-09 00:55:50 on dist-test-slave-j2vl"
server_key: "a265abbb03687c17dc2780763677f18b"
server_key_iv: "27aa15c0c2b0dcdf76e98ffabfeba0cc"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:50.379390 30634 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20260809 00:55:50.401470 30655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:50.403426 30634 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260809 00:55:50.403710 30634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "1214beb866ba4414857f3476898ca05f"
format_stamp: "Formatted at 2026-08-09 00:55:50 on dist-test-slave-j2vl"
server_key: "a265abbb03687c17dc2780763677f18b"
server_key_iv: "27aa15c0c2b0dcdf76e98ffabfeba0cc"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:50.404023 30634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:55:50.471655 30634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:55:50.477695 30634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:55:50.478237 30634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:55:50.479806 30634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:55:50.482151 30634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:55:50.482312 30634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:50.482525 30634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:55:50.482627 30634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:50.562497 30634 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:44073
I20260809 00:55:50.562593 30838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:44073 every 8 connection(s)
I20260809 00:55:50.564342 30634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:55:50.570276 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 30634
I20260809 00:55:50.570480 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:55:50.571362 10116 external_mini_cluster.cc:1428] Setting key 884f81912942563df60daa5c1c5ddba1
Aug 09 00:55:50 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236950, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:55:50.601111 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:50.568045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:35667 (local address 127.9.225.62:35523)
0809 00:55:50.568999 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:50.569005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:50.571661 (+ 2656us) server_negotiation.cc:407] Connection header received
0809 00:55:50.575469 (+ 3808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:50.575478 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:50.575563 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:50.575664 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:50.578090 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:50.578989 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:50.581278 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:50.582149 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:50.582314 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:50.592568 (+ 10254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:50.592579 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:50.592587 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:50.592654 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:50.597006 (+ 4352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:50.598242 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:50.598247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:50.598250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:50.598472 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:50.599091 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:50.599095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:50.599097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:50.599479 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:50.599619 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:50.600545 (+ 926us) server_negotiation.cc:299] Negotiation successful
0809 00:55:50.600770 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":544,"threads_started":1}
I20260809 00:55:50.602602 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:50.568884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35523 (local address 127.9.225.3:35667)
0809 00:55:50.571411 (+ 2527us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:50.571473 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:50.575057 (+ 3584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:50.576045 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:50.576055 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:50.576869 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:55:50.577873 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:50.577897 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:50.579181 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:50.579185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:50.581068 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:50.581116 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:50.582700 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:50.582727 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:50.583278 (+ 551us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:50.584512 (+ 1234us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:55:50.584546 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:55:50.589112 (+ 4566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:55:50.597233 (+ 8121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:50.597244 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:50.597264 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:50.598065 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:50.598600 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:50.598603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:50.598606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:50.598968 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:50.599661 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:55:50.599669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:55:50.600071 (+ 402us) client_negotiation.cc:769] Sending connection context
0809 00:55:50.600385 (+ 314us) client_negotiation.cc:240] Negotiation successful
0809 00:55:50.600978 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2014,"thread_start_us":1592,"threads_started":1}
I20260809 00:55:50.604537 30840 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35523
I20260809 00:55:50.604962 30840 heartbeater.cc:464] Registering TS with master...
I20260809 00:55:50.606117 30840 heartbeater.cc:511] Master 127.9.225.62:35523 requested a full tablet report, sending...
I20260809 00:55:50.608045 30039 ts_manager.cc:194] Registered new tserver with Master: 1214beb866ba4414857f3476898ca05f (127.9.225.3:44073)
I20260809 00:55:50.609141 30039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:35667
I20260809 00:55:50.621091 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:55:50.623121 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 30158
Aug 09 00:55:50 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:55:50.648043 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:50.624788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.9.225.1:33633)
0809 00:55:50.626284 (+ 1496us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:50.626294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:50.626324 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:55:50.626393 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:50.626399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:50.626712 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:50.626886 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:50.629434 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:50.630966 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:50.633607 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:50.634441 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:50.634566 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:50.640413 (+ 5847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:50.640433 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:50.640453 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:50.640520 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:50.644421 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:50.645412 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:50.645429 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:50.645432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:50.645669 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:50.646272 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:50.646276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:50.646278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:50.646739 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:50.647134 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:50.647361 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 00:55:50.647607 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1228,"thread_start_us":851,"threads_started":1}
I20260809 00:55:50.649688 30387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:55:50.740010 30627 heartbeater.cc:503] Master 127.9.225.62:35523 was elected leader, sending a full tablet report...
I20260809 00:55:50.740702 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 30158
I20260809 00:55:50.769586 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 30408
Aug 09 00:55:50 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:55:50.794888 30852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:50.775488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.9.225.2:40699)
0809 00:55:50.776414 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:50.776423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:50.776453 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:55:50.776533 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:50.776541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:50.776779 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:50.777002 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:50.778829 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:50.780012 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:50.781951 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:50.782587 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:50.782756 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:50.787888 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:50.787907 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:50.787926 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:50.787974 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:50.791279 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:50.792238 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:50.792248 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:50.792250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:50.792441 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:50.792916 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:50.792921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:50.792923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:50.793442 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:50.793792 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:50.794134 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 00:55:50.794457 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":453,"threads_started":1}
I20260809 00:55:50.796420 30616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:55:50.883852 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 30408
I20260809 00:55:50.915781 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 30634
Aug 09 00:55:50 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:55:50.936619 30856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:50.916910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51474 (local address 127.9.225.3:44073)
0809 00:55:50.917550 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:50.917558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:50.917592 (+ 34us) server_negotiation.cc:407] Connection header received
0809 00:55:50.917659 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:50.917689 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:50.918018 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:50.918204 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:50.919759 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:50.920837 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:50.922407 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:50.923010 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:50.923168 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:50.929060 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:50.929081 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:50.929101 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:50.929151 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:50.932700 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:50.933795 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:50.933807 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:50.933809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:50.933972 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:50.934651 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:50.934656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:50.934658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:50.935128 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:50.935493 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:50.935927 (+ 434us) server_negotiation.cc:299] Negotiation successful
0809 00:55:50.936247 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":280,"threads_started":1}
I20260809 00:55:50.938179 30833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:55:51.027128 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 30634
I20260809 00:55:51.060907 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 29992
I20260809 00:55:51.062506 30087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:55:51.174854 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 29992
I20260809 00:55:51.217093 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35523
--webserver_interface=127.9.225.62
--webserver_port=40301
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:35523
--ranger_config_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.225.20:33065/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.225.20:33065/kms with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:55:51.593924 30859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:55:51.594273 30859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:55:51.594374 30859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:55:51.602214 30859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:55:51.602361 30859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:55:51.602409 30859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:55:51.602468 30859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:55:51.602520 30859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:55:51.613452 30859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.225.20:33065/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:35523
--ranger_config_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:55:51.614954 30859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:55:51.616369 30859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:55:51.629899 30868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:51.629916 30867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:51.630911 30873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:51.630861 30859 server_base.cc:1034] running on GCE node
I20260809 00:55:51.632175 30859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:55:51.634483 30859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:55:51.635771 30859 hybrid_clock.cc:648] HybridClock initialized: now 1786236951635693 us; error 85 us; skew 500 ppm
I20260809 00:55:51.636274 30859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:55:51 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236951, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:51.640775 30859 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:55:51.643083 30859 webserver.cc:460] Webserver started at http://127.9.225.62:40301/ using document root <none> and password file <none>
I20260809 00:55:51.644174 30859 fs_manager.cc:359] Metadata directory not provided
I20260809 00:55:51.644387 30859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 00:55:51 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236951, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:55:51.692137 30859 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 00:55:51.696108 30882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:51.697986 30859 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260809 00:55:51.698249 30859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "8e31e32134c74e288f0398c883eefad2"
format_stamp: "Formatted at 2026-08-09 00:55:46 on dist-test-slave-j2vl"
server_key: "8850178fb033bf0f961090e8ba15ffe2"
server_key_iv: "637a88c8238615a2a86c421368a07b83"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:51.699424 30859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:55:51.732849 30859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:55:51.737759 30859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:55:51.738133 30859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:55:51.759255 30859 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:35523
I20260809 00:55:51.759268 30972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:35523 every 8 connection(s)
I20260809 00:55:51.761080 30859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:55:51.769137 30973 sys_catalog.cc:263] Verifying existing consensus state
I20260809 00:55:51.771086 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 30859
I20260809 00:55:51.772037 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.225.20:33065/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:33633
--local_ip_for_outbound_sockets=127.9.225.1
--tserver_master_addrs=127.9.225.62:35523
--webserver_port=33003
--webserver_interface=127.9.225.1
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.225.20:33065/kms with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:55:51.773102 30973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2: Bootstrap starting.
I20260809 00:55:51.797566 30973 log.cc:824] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2: Log is configured to *not* fsync() on all Append() calls
I20260809 00:55:51.812703 30973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 00:55:51.813522 30973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2: Bootstrap complete.
I20260809 00:55:51.828528 30973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e31e32134c74e288f0398c883eefad2" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35523 } }
I20260809 00:55:51.829926 30973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e31e32134c74e288f0398c883eefad2, State: Initialized, Role: FOLLOWER
I20260809 00:55:51.830945 30973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e31e32134c74e288f0398c883eefad2" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35523 } }
I20260809 00:55:51.831286 30973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:55:51.831487 30973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:55:51.831696 30973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 1 FOLLOWER]: Advancing to term 2
I20260809 00:55:51.837919 30973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e31e32134c74e288f0398c883eefad2" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35523 } }
I20260809 00:55:51.838675 30973 leader_election.cc:302] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e31e32134c74e288f0398c883eefad2; no voters:
I20260809 00:55:51.839432 30973 leader_election.cc:288] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 00:55:51.839627 30977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 00:55:51.840891 30977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [term 2 LEADER]: Becoming Leader. State: Replica: 8e31e32134c74e288f0398c883eefad2, State: Running, Role: LEADER
I20260809 00:55:51.841634 30977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e31e32134c74e288f0398c883eefad2" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35523 } }
I20260809 00:55:51.841903 30973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:55:51.844491 30979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e31e32134c74e288f0398c883eefad2. Latest consensus state: current_term: 2 leader_uuid: "8e31e32134c74e288f0398c883eefad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e31e32134c74e288f0398c883eefad2" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35523 } } }
I20260809 00:55:51.844955 30979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [sys.catalog]: This master's current role is: LEADER
I20260809 00:55:51.846637 30978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8e31e32134c74e288f0398c883eefad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e31e32134c74e288f0398c883eefad2" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35523 } } }
I20260809 00:55:51.847064 30978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2 [sys.catalog]: This master's current role is: LEADER
I20260809 00:55:51.856454 30983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:55:51.863402 30983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:55:51.865195 30983 catalog_manager.cc:1259] Loaded cluster ID: 4725c7067dd5453188e30df5bd88fc4c
I20260809 00:55:51.865430 30983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:55:51.869832 30983 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:55:51.871948 30983 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2: Loaded TSK: 0
I20260809 00:55:51.874164 30983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:55:51.879428 30973 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260809 00:55:52.313400 30975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:55:52.313786 30975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:55:52.314021 30975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:55:52.325910 30975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:55:52.326088 30975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:55:52.326288 30975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:55:52.341769 30975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.225.20:33065/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:33633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=33003
--tserver_master_addrs=127.9.225.62:35523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:55:52.343659 30975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:55:52.345489 30975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:55:52.377408 31030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:52.378408 31031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:52.379649 30975 server_base.cc:1034] running on GCE node
W20260809 00:55:52.379877 31035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:52.380740 30975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:55:52.389797 30975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:55:52.391490 30975 hybrid_clock.cc:648] HybridClock initialized: now 1786236952391399 us; error 94 us; skew 500 ppm
I20260809 00:55:52.392060 30975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:55:52 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236952, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:52.397161 30975 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:55:52.418769 30975 webserver.cc:460] Webserver started at http://127.9.225.1:33003/ using document root <none> and password file <none>
I20260809 00:55:52.420112 30975 fs_manager.cc:359] Metadata directory not provided
I20260809 00:55:52.420414 30975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 00:55:52 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236952, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:55:52.474457 30975 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.004s sys 0.002s
I20260809 00:55:52.479107 31047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:52.481245 30975 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260809 00:55:52.481617 30975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "5ee8eec859b940dead68aac2c0f5f8f4"
format_stamp: "Formatted at 2026-08-09 00:55:48 on dist-test-slave-j2vl"
server_key: "655c5ef0f7a3e879376c9541907143ef"
server_key_iv: "cace6fd2e3c7b507f75c1c6b47b5a9fc"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:52.483250 30975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:55:52.516000 30975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:55:52.521240 30975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:55:52.521723 30975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:55:52.526562 30975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:55:52.528657 30975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:55:52.528815 30975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:52.528986 30975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:55:52.529081 30975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:52.584146 30975 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:33633
I20260809 00:55:52.584224 31233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:33633 every 8 connection(s)
I20260809 00:55:52.586493 30975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:55:52.594098 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 30975
I20260809 00:55:52.595098 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.225.20:33065/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:40699
--local_ip_for_outbound_sockets=127.9.225.2
--tserver_master_addrs=127.9.225.62:35523
--webserver_port=43719
--webserver_interface=127.9.225.2
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.225.20:33065/kms with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:55:52 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236952, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:55:52.623456 31237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:52.590484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35523 (local address 127.9.225.1:36529)
0809 00:55:52.591783 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:52.591835 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:52.594107 (+ 2272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:52.597395 (+ 3288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:52.597410 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:52.598139 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:55:52.599048 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:52.599074 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:52.601580 (+ 2506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:52.601586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:52.603854 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:52.603877 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:52.606398 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:52.606409 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:52.606750 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:52.607719 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:55:52.607757 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:55:52.611186 (+ 3429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:55:52.616937 (+ 5751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:52.616952 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:52.616973 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:52.617795 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:52.618522 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:52.618530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:52.618537 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:52.618885 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:52.620171 (+ 1286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:55:52.620182 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:55:52.620665 (+ 483us) client_negotiation.cc:769] Sending connection context
0809 00:55:52.621056 (+ 391us) client_negotiation.cc:240] Negotiation successful
0809 00:55:52.621799 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":800,"threads_started":1}
I20260809 00:55:52.623687 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:52.590346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:36529 (local address 127.9.225.62:35523)
0809 00:55:52.591293 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:52.591315 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:52.594625 (+ 3310us) server_negotiation.cc:407] Connection header received
0809 00:55:52.595253 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:52.595279 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:52.595912 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:52.596425 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:52.599318 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:52.601255 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:52.604181 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:52.605613 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:52.605862 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:52.611669 (+ 5807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:52.611686 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:52.611714 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:52.611797 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:52.616632 (+ 4835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:52.618020 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:52.618034 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:52.618045 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:52.618350 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:52.619037 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:52.619044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:52.619050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:52.619848 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:52.620068 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:52.621780 (+ 1712us) server_negotiation.cc:299] Negotiation successful
0809 00:55:52.622283 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":331,"threads_started":1}
I20260809 00:55:52.628504 31234 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35523
I20260809 00:55:52.628924 31234 heartbeater.cc:464] Registering TS with master...
I20260809 00:55:52.629890 31234 heartbeater.cc:511] Master 127.9.225.62:35523 requested a full tablet report, sending...
W20260809 00:55:52.634235 31234 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:35523 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 00:55:52.634557 31234 heartbeater.cc:464] Registering TS with master...
I20260809 00:55:52.634972 31234 heartbeater.cc:511] Master 127.9.225.62:35523 requested a full tablet report, sending...
I20260809 00:55:52.637127 31234 heartbeater.cc:464] Registering TS with master...
I20260809 00:55:52.637575 31234 heartbeater.cc:511] Master 127.9.225.62:35523 requested a full tablet report, sending...
W20260809 00:55:52.639204 31234 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 00:55:53.067823 31240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:55:53.068253 31240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:55:53.068437 31240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:55:53.076403 31240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:55:53.076619 31240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:55:53.076805 31240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:55:53.093050 31240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.225.20:33065/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:40699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=43719
--tserver_master_addrs=127.9.225.62:35523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:55:53.095361 31240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:55:53.097472 31240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:55:53.131317 31265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:53.131386 31264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:53.134300 31269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:53.135267 31240 server_base.cc:1034] running on GCE node
I20260809 00:55:53.136232 31240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:55:53.145836 31240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:55:53.147320 31240 hybrid_clock.cc:648] HybridClock initialized: now 1786236953147226 us; error 100 us; skew 500 ppm
I20260809 00:55:53.147886 31240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:55:53 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:53.152793 31240 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:55:53.167491 31240 webserver.cc:460] Webserver started at http://127.9.225.2:43719/ using document root <none> and password file <none>
I20260809 00:55:53.168680 31240 fs_manager.cc:359] Metadata directory not provided
I20260809 00:55:53.168884 31240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 00:55:53 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:55:53.214962 31240 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.004s sys 0.012s
Aug 09 00:55:53 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786236953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:53.227308 31282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:53.229861 31240 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.008s
I20260809 00:55:53.230191 31240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "efab3b7a9a9942efafb119e28faa0642"
format_stamp: "Formatted at 2026-08-09 00:55:49 on dist-test-slave-j2vl"
server_key: "b157a175502b757408d9949e81dba97c"
server_key_iv: "c4b74676485f430dc40740f0ea38a36e"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:53.231948 31240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:55:53.301667 31240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:55:53.308041 31240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:55:53.308404 31240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:55:53.309767 31240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:55:53.311605 31240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:55:53.311744 31240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:53.311902 31240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:55:53.311996 31240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:53.375236 31240 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:40699
I20260809 00:55:53.375274 31476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:40699 every 8 connection(s)
I20260809 00:55:53.378422 31240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:55:53.388226 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 31240
I20260809 00:55:53.389205 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.225.20:33065/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:44073
--local_ip_for_outbound_sockets=127.9.225.3
--tserver_master_addrs=127.9.225.62:35523
--webserver_port=41333
--webserver_interface=127.9.225.3
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.225.20:33065/kms with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:55:53 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:55:53.413718 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:53.384231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:38729 (local address 127.9.225.62:35523)
0809 00:55:53.384956 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:53.384963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:53.387337 (+ 2374us) server_negotiation.cc:407] Connection header received
0809 00:55:53.390278 (+ 2941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:53.390286 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:53.390456 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:53.390586 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:53.393568 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:53.394826 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:53.397756 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:53.398459 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:53.398594 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:53.404569 (+ 5975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:53.404578 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:53.404583 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:53.404641 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:53.408772 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:53.410138 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:53.410145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:53.410149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:53.410451 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:53.411249 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:53.411254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:53.411258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:53.411664 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:53.411817 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:53.412986 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0809 00:55:53.413277 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":385,"threads_started":1}
I20260809 00:55:53.416675 31481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:53.385333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35523 (local address 127.9.225.2:38729)
0809 00:55:53.387080 (+ 1747us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:53.387146 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:53.389806 (+ 2660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:53.391197 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:53.391213 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:53.392050 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:55:53.393133 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:53.393169 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:53.395109 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:53.395115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:53.397526 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:53.397548 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:53.399082 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:53.399091 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:53.399464 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:53.400445 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:55:53.400482 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:55:53.404154 (+ 3672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:55:53.409060 (+ 4906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:53.409072 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:53.409097 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:53.409944 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:53.410673 (+ 729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:53.410679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:53.410682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:53.411067 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:53.411944 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:55:53.411954 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:55:53.412464 (+ 510us) client_negotiation.cc:769] Sending connection context
0809 00:55:53.414369 (+ 1905us) client_negotiation.cc:240] Negotiation successful
0809 00:55:53.415114 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"thread_start_us":605,"threads_started":1}
I20260809 00:55:53.419266 31477 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35523
I20260809 00:55:53.419675 31477 heartbeater.cc:464] Registering TS with master...
I20260809 00:55:53.420781 31477 heartbeater.cc:511] Master 127.9.225.62:35523 requested a full tablet report, sending...
I20260809 00:55:53.426275 30916 ts_manager.cc:194] Registered new tserver with Master: efab3b7a9a9942efafb119e28faa0642 (127.9.225.2:40699)
I20260809 00:55:53.428815 30916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:38729
I20260809 00:55:53.639371 31234 heartbeater.cc:464] Registering TS with master...
I20260809 00:55:53.639900 31234 heartbeater.cc:511] Master 127.9.225.62:35523 requested a full tablet report, sending...
I20260809 00:55:53.641405 30916 ts_manager.cc:194] Registered new tserver with Master: 5ee8eec859b940dead68aac2c0f5f8f4 (127.9.225.1:33633)
I20260809 00:55:53.642704 30916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:36529
Aug 09 00:55:53 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
W20260809 00:55:53.884523 31483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:55:53.884974 31483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:55:53.885174 31483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:55:53.893873 31483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:55:53.894069 31483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:55:53.894302 31483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:55:53.905592 31483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.225.20:33065/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:44073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=41333
--tserver_master_addrs=127.9.225.62:35523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:55:53.907207 31483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:55:53.908624 31483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:55:53.929419 31493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:53.940091 31483 server_base.cc:1034] running on GCE node
W20260809 00:55:53.940390 31496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:53.940620 31494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:53.941625 31483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:55:53.943033 31483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:55:53.944336 31483 hybrid_clock.cc:648] HybridClock initialized: now 1786236953944266 us; error 55 us; skew 500 ppm
I20260809 00:55:53.944779 31483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:55:53 dist-test-slave-j2vl krb5kdc[29254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:53.949471 31483 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:55:53.959223 31483 webserver.cc:460] Webserver started at http://127.9.225.3:41333/ using document root <none> and password file <none>
I20260809 00:55:53.960397 31483 fs_manager.cc:359] Metadata directory not provided
I20260809 00:55:53.960614 31483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 00:55:53 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:55:53.986248 31483 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:55:53.990862 31506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:53.993112 31483 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260809 00:55:53.993417 31483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "1214beb866ba4414857f3476898ca05f"
format_stamp: "Formatted at 2026-08-09 00:55:50 on dist-test-slave-j2vl"
server_key: "a265abbb03687c17dc2780763677f18b"
server_key_iv: "27aa15c0c2b0dcdf76e98ffabfeba0cc"
server_key_version: "kuduclusterkey@0"
I20260809 00:55:53.995090 31483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:55:54.017483 31483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:55:54.023692 31483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:55:54.024130 31483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:55:54.025611 31483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:55:54.027756 31483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:55:54.027925 31483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 00:55:54.028093 31483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:55:54.028213 31483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:54.118781 31683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:44073 every 8 connection(s)
I20260809 00:55:54.119035 31483 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:44073
I20260809 00:55:54.121233 31483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
Aug 09 00:55:54 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for HTTP/127.9.225.20@KRBTEST.COM
I20260809 00:55:54.124925 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 31483
I20260809 00:55:54.196645 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.159730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.9.225.62:35523)
0809 00:55:54.169379 (+ 9649us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:54.169386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:54.169412 (+ 26us) server_negotiation.cc:407] Connection header received
0809 00:55:54.169506 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:54.169512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:54.169629 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:54.169774 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:54.173320 (+ 3546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.174448 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.177104 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.177749 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.177901 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.181921 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:54.181928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:54.181932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:54.181986 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:54.187725 (+ 5739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:54.192281 (+ 4556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:54.192289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:54.192292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:54.192546 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:54.193847 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:54.193852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:54.193856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:54.194236 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:54.194558 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:54.195941 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0809 00:55:54.196237 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9459,"thread_start_us":10830,"threads_started":1}
Aug 09 00:55:54 dist-test-slave-j2vl krb5kdc[29254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236953, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:55:54.214675 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.126242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:40419 (local address 127.9.225.62:35523)
0809 00:55:54.159520 (+ 33278us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:54.159527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:54.166551 (+ 7024us) server_negotiation.cc:407] Connection header received
0809 00:55:54.176530 (+ 9979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:54.176537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:54.176723 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:54.176823 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:54.187998 (+ 11175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.189325 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.192757 (+ 3432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.193503 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.193615 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.205199 (+ 11584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:54.205208 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:54.205214 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:54.205276 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:54.209129 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:54.211396 (+ 2267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:54.211403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:54.211408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:54.211590 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:54.212229 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:54.212234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:54.212237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:54.212631 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:54.212762 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:54.213949 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0809 00:55:54.214253 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33109,"thread_start_us":33303,"threads_started":1}
I20260809 00:55:54.218122 31689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.125389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35523 (local address 127.9.225.3:40419)
0809 00:55:54.162563 (+ 37174us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:54.162636 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:54.174834 (+ 12198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:54.178744 (+ 3910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:54.178764 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:54.179642 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:55:54.180757 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:54.180794 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.189645 (+ 8851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.189652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:54.192058 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:54.192085 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.194893 (+ 2808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.194906 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:54.195304 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.200579 (+ 5275us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:55:54.200623 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:55:54.204721 (+ 4098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:55:54.210409 (+ 5688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:54.210422 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:54.210448 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:54.211203 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:54.211760 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:54.211765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:54.211769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:54.212084 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:54.212889 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:55:54.212899 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:55:54.213435 (+ 536us) client_negotiation.cc:769] Sending connection context
0809 00:55:54.215631 (+ 2196us) client_negotiation.cc:240] Negotiation successful
0809 00:55:54.216329 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36701,"thread_start_us":39795,"threads_started":1}
I20260809 00:55:54.220424 31684 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35523
I20260809 00:55:54.220803 31684 heartbeater.cc:464] Registering TS with master...
I20260809 00:55:54.221863 31684 heartbeater.cc:511] Master 127.9.225.62:35523 requested a full tablet report, sending...
I20260809 00:55:54.223987 30916 ts_manager.cc:194] Registered new tserver with Master: 1214beb866ba4414857f3476898ca05f (127.9.225.3:44073)
I20260809 00:55:54.225158 30916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:40419
I20260809 00:55:54.230993 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:55:54.354552 31691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.331623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.9.225.62:35523)
0809 00:55:54.331864 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:54.331871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:54.334749 (+ 2878us) server_negotiation.cc:407] Connection header received
0809 00:55:54.335139 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:54.335146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:54.335252 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:54.335339 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:54.338169 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.339264 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.341551 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.342259 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.342424 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.346495 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:54.346504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:54.346510 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:54.346571 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:54.350619 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:54.351756 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:54.351763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:54.351767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:54.351950 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:54.352465 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:54.352471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:54.352474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:54.352882 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:54.353138 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:54.353851 (+ 713us) server_negotiation.cc:299] Negotiation successful
0809 00:55:54.354144 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 00:55:54.361918 30916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:55:54.374708 30916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:55:54.434640 31477 heartbeater.cc:503] Master 127.9.225.62:35523 was elected leader, sending a full tablet report...
I20260809 00:55:54.446384 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.407054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:40699 (local address 127.0.0.1:33114)
0809 00:55:54.422916 (+ 15862us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:54.422980 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:54.423206 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:54.433917 (+ 10711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:54.433929 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:54.433980 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:55:54.435949 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:54.435969 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.438160 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.438166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:54.440925 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:54.440945 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.445135 (+ 4190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.445143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:54.445427 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.445512 (+ 85us) client_negotiation.cc:769] Sending connection context
0809 00:55:54.445656 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 00:55:54.445913 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15480,"thread_start_us":15749,"threads_started":1}
I20260809 00:55:54.449702 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.407160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.9.225.2:40699)
0809 00:55:54.432816 (+ 25656us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:54.432825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:54.432862 (+ 37us) server_negotiation.cc:407] Connection header received
0809 00:55:54.432989 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:54.432999 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:54.433308 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:54.435516 (+ 2208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:55:54.436208 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.437962 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.441143 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.442797 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.447123 (+ 4326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.447442 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:54.447538 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 00:55:54.447742 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25462,"thread_start_us":26397,"threads_started":1}
I20260809 00:55:54.454056 31706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.410412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:33633 (local address 127.0.0.1:48076)
0809 00:55:54.433841 (+ 23429us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:54.433865 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:54.434119 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:54.435197 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:54.435201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:54.435232 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:55:54.437292 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:54.437308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.446436 (+ 9128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.446444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:54.450348 (+ 3904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:54.450368 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.453099 (+ 2731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.453107 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:54.453311 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.453370 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 00:55:54.453468 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 00:55:54.453640 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12225,"mutex_wait_us":1,"thread_start_us":13007,"threads_started":1}
I20260809 00:55:54.455519 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.424250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.9.225.3:44073)
0809 00:55:54.433902 (+ 9652us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:54.433912 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:54.441904 (+ 7992us) server_negotiation.cc:407] Connection header received
0809 00:55:54.442098 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:54.442112 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:54.442457 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:54.442701 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:55:54.443701 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.445265 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.451874 (+ 6609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.453561 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.453682 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.453924 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:54.454923 (+ 999us) server_negotiation.cc:299] Negotiation successful
0809 00:55:54.455137 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9446,"thread_start_us":9280,"threads_started":1}
I20260809 00:55:54.454875 31371 tablet_service.cc:1467] Processing CreateTablet for tablet 1971aa9c4ded48abbac3eddd7e3a4103 (DEFAULT_TABLE table=test-table [id=7e775bc0cd38470192eea85f4aa8bf1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:55:54.456681 31713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.423921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:44073 (local address 127.0.0.1:51476)
0809 00:55:54.441743 (+ 17822us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:54.441769 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:54.441924 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:54.443067 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:54.443072 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:54.443101 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:55:54.443488 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:54.443502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.445483 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.445488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:54.451620 (+ 6132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:54.451641 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.454432 (+ 2791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.454443 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:54.454636 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.454678 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 00:55:54.456099 (+ 1421us) client_negotiation.cc:240] Negotiation successful
0809 00:55:54.456264 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17660,"mutex_wait_us":923,"thread_start_us":8953,"threads_started":1}
I20260809 00:55:54.456724 31371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1971aa9c4ded48abbac3eddd7e3a4103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:55:54.458853 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.410669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.9.225.1:33633)
0809 00:55:54.432329 (+ 21660us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:54.432340 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:54.433981 (+ 1641us) server_negotiation.cc:407] Connection header received
0809 00:55:54.434338 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:54.434348 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:54.434608 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:54.434869 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:55:54.443023 (+ 8154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.444761 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.450589 (+ 5828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.452269 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.452438 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.452710 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:54.458210 (+ 5500us) server_negotiation.cc:299] Negotiation successful
0809 00:55:54.458502 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21037,"thread_start_us":21247,"threads_started":1}
I20260809 00:55:54.465885 31122 tablet_service.cc:1467] Processing CreateTablet for tablet 1971aa9c4ded48abbac3eddd7e3a4103 (DEFAULT_TABLE table=test-table [id=7e775bc0cd38470192eea85f4aa8bf1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:55:54.466974 31581 tablet_service.cc:1467] Processing CreateTablet for tablet 1971aa9c4ded48abbac3eddd7e3a4103 (DEFAULT_TABLE table=test-table [id=7e775bc0cd38470192eea85f4aa8bf1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:55:54.468814 31122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1971aa9c4ded48abbac3eddd7e3a4103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:55:54.469904 31581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1971aa9c4ded48abbac3eddd7e3a4103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:55:54.490268 31714 tablet_bootstrap.cc:492] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642: Bootstrap starting.
I20260809 00:55:54.494966 31714 tablet_bootstrap.cc:654] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642: Neither blocks nor log segments found. Creating new log.
I20260809 00:55:54.496682 31714 log.cc:824] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642: Log is configured to *not* fsync() on all Append() calls
I20260809 00:55:54.508761 31717 tablet_bootstrap.cc:492] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4: Bootstrap starting.
I20260809 00:55:54.515503 31717 tablet_bootstrap.cc:654] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4: Neither blocks nor log segments found. Creating new log.
I20260809 00:55:54.517498 31717 log.cc:824] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4: Log is configured to *not* fsync() on all Append() calls
I20260809 00:55:54.518260 31715 tablet_bootstrap.cc:492] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f: Bootstrap starting.
I20260809 00:55:54.521677 31714 tablet_bootstrap.cc:492] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642: No bootstrap required, opened a new log
I20260809 00:55:54.522148 31714 ts_tablet_manager.cc:1397] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.005s
I20260809 00:55:54.529304 31717 tablet_bootstrap.cc:492] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4: No bootstrap required, opened a new log
I20260809 00:55:54.529486 31715 tablet_bootstrap.cc:654] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f: Neither blocks nor log segments found. Creating new log.
I20260809 00:55:54.529943 31717 ts_tablet_manager.cc:1397] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.006s
I20260809 00:55:54.531062 31715 log.cc:824] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:55:54.541502 31717 raft_consensus.cc:348] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efab3b7a9a9942efafb119e28faa0642" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40699 } } peers { permanent_uuid: "1214beb866ba4414857f3476898ca05f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 44073 } } peers { permanent_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33633 } }
I20260809 00:55:54.542279 31717 raft_consensus.cc:374] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:55:54.541136 31714 raft_consensus.cc:348] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efab3b7a9a9942efafb119e28faa0642" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40699 } } peers { permanent_uuid: "1214beb866ba4414857f3476898ca05f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 44073 } } peers { permanent_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33633 } }
I20260809 00:55:54.542492 31717 raft_consensus.cc:729] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee8eec859b940dead68aac2c0f5f8f4, State: Initialized, Role: FOLLOWER
I20260809 00:55:54.542624 31714 raft_consensus.cc:374] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:55:54.542764 31714 raft_consensus.cc:729] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efab3b7a9a9942efafb119e28faa0642, State: Initialized, Role: FOLLOWER
I20260809 00:55:54.543697 31714 consensus_queue.cc:260] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efab3b7a9a9942efafb119e28faa0642" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40699 } } peers { permanent_uuid: "1214beb866ba4414857f3476898ca05f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 44073 } } peers { permanent_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33633 } }
I20260809 00:55:54.543704 31717 consensus_queue.cc:260] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efab3b7a9a9942efafb119e28faa0642" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40699 } } peers { permanent_uuid: "1214beb866ba4414857f3476898ca05f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 44073 } } peers { permanent_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33633 } }
I20260809 00:55:54.545404 31715 tablet_bootstrap.cc:492] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f: No bootstrap required, opened a new log
I20260809 00:55:54.545840 31715 ts_tablet_manager.cc:1397] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.006s
I20260809 00:55:54.555353 31717 ts_tablet_manager.cc:1428] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4: Time spent starting tablet: real 0.025s user 0.011s sys 0.011s
I20260809 00:55:54.555347 31234 heartbeater.cc:503] Master 127.9.225.62:35523 was elected leader, sending a full tablet report...
I20260809 00:55:54.557723 31714 ts_tablet_manager.cc:1428] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642: Time spent starting tablet: real 0.035s user 0.013s sys 0.014s
I20260809 00:55:54.560693 31715 raft_consensus.cc:348] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efab3b7a9a9942efafb119e28faa0642" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40699 } } peers { permanent_uuid: "1214beb866ba4414857f3476898ca05f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 44073 } } peers { permanent_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33633 } }
I20260809 00:55:54.562211 31715 raft_consensus.cc:374] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:55:54.562430 31715 raft_consensus.cc:729] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1214beb866ba4414857f3476898ca05f, State: Initialized, Role: FOLLOWER
I20260809 00:55:54.563478 31715 consensus_queue.cc:260] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efab3b7a9a9942efafb119e28faa0642" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40699 } } peers { permanent_uuid: "1214beb866ba4414857f3476898ca05f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 44073 } } peers { permanent_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33633 } }
I20260809 00:55:54.564834 31684 heartbeater.cc:503] Master 127.9.225.62:35523 was elected leader, sending a full tablet report...
I20260809 00:55:54.565721 31715 ts_tablet_manager.cc:1428] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
W20260809 00:55:54.598708 31235 tablet.cc:2369] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:55:54.631938 31478 tablet.cc:2369] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:55:54.660264 31727 raft_consensus.cc:482] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:55:54.660614 31727 raft_consensus.cc:504] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efab3b7a9a9942efafb119e28faa0642" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40699 } } peers { permanent_uuid: "1214beb866ba4414857f3476898ca05f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 44073 } } peers { permanent_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33633 } }
I20260809 00:55:54.662763 31727 leader_election.cc:288] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efab3b7a9a9942efafb119e28faa0642 (127.9.225.2:40699), 1214beb866ba4414857f3476898ca05f (127.9.225.3:44073)
I20260809 00:55:54.671998 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.663284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:55733 (local address 127.9.225.2:40699)
0809 00:55:54.663498 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:54.663504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:54.664133 (+ 629us) server_negotiation.cc:407] Connection header received
0809 00:55:54.664469 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:54.664479 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:54.664582 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:54.664653 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:55:54.665475 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.666502 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.669308 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.670635 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.670786 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.670916 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:54.671539 (+ 623us) server_negotiation.cc:299] Negotiation successful
0809 00:55:54.671688 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 00:55:54.671964 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.664359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:39915 (local address 127.9.225.3:44073)
0809 00:55:54.664625 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:54.664632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:54.664981 (+ 349us) server_negotiation.cc:407] Connection header received
0809 00:55:54.665229 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:54.665236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:54.665312 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:54.665447 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:55:54.665977 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.666933 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.669331 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.670635 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.670786 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.670906 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:54.671538 (+ 632us) server_negotiation.cc:299] Negotiation successful
0809 00:55:54.671688 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 00:55:54.672312 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.664196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:44073 (local address 127.9.225.1:39915)
0809 00:55:54.664848 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:54.664865 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:54.665076 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:54.665474 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:54.665479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:54.665504 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:55:54.665814 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:54.665822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.667081 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.667086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:54.669187 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:54.669201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.670994 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.671002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:54.671322 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.671377 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 00:55:54.671494 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 00:55:54.671744 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":99,"thread_start_us":318,"threads_started":1}
I20260809 00:55:54.672349 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.663217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:40699 (local address 127.9.225.1:55733)
0809 00:55:54.664003 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:54.664026 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:54.664186 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:54.664886 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:54.664890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:54.664917 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:55:54.665276 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:54.665284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.666688 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.666693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:54.669067 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:54.669092 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.670994 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.671002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:54.671334 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.671381 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 00:55:54.671494 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 00:55:54.671744 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":160,"spinlock_wait_cycles":21248,"thread_start_us":443,"threads_started":1}
I20260809 00:55:54.673542 31403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1971aa9c4ded48abbac3eddd7e3a4103" candidate_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efab3b7a9a9942efafb119e28faa0642" is_pre_election: true
I20260809 00:55:54.673547 31610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1971aa9c4ded48abbac3eddd7e3a4103" candidate_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1214beb866ba4414857f3476898ca05f" is_pre_election: true
I20260809 00:55:54.674222 31610 raft_consensus.cc:2454] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee8eec859b940dead68aac2c0f5f8f4 in term 0.
I20260809 00:55:54.674224 31403 raft_consensus.cc:2454] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee8eec859b940dead68aac2c0f5f8f4 in term 0.
I20260809 00:55:54.675036 31053 leader_election.cc:302] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1214beb866ba4414857f3476898ca05f, 5ee8eec859b940dead68aac2c0f5f8f4; no voters:
I20260809 00:55:54.675511 31727 raft_consensus.cc:2781] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:55:54.675670 31727 raft_consensus.cc:482] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:55:54.675791 31727 raft_consensus.cc:3037] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:55:54.678282 31727 raft_consensus.cc:504] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efab3b7a9a9942efafb119e28faa0642" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40699 } } peers { permanent_uuid: "1214beb866ba4414857f3476898ca05f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 44073 } } peers { permanent_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33633 } }
I20260809 00:55:54.678931 31727 leader_election.cc:288] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [CANDIDATE]: Term 1 election: Requested vote from peers efab3b7a9a9942efafb119e28faa0642 (127.9.225.2:40699), 1214beb866ba4414857f3476898ca05f (127.9.225.3:44073)
I20260809 00:55:54.679680 31403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1971aa9c4ded48abbac3eddd7e3a4103" candidate_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efab3b7a9a9942efafb119e28faa0642"
I20260809 00:55:54.679762 31610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1971aa9c4ded48abbac3eddd7e3a4103" candidate_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1214beb866ba4414857f3476898ca05f"
I20260809 00:55:54.679925 31403 raft_consensus.cc:3037] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:55:54.680042 31610 raft_consensus.cc:3037] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f [term 0 FOLLOWER]: Advancing to term 1
W20260809 00:55:54.682382 31685 tablet.cc:2369] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:55:54.682808 31403 raft_consensus.cc:2454] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee8eec859b940dead68aac2c0f5f8f4 in term 1.
I20260809 00:55:54.683077 31610 raft_consensus.cc:2454] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee8eec859b940dead68aac2c0f5f8f4 in term 1.
I20260809 00:55:54.683609 31053 leader_election.cc:302] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1214beb866ba4414857f3476898ca05f, 5ee8eec859b940dead68aac2c0f5f8f4; no voters:
I20260809 00:55:54.683955 31727 raft_consensus.cc:2781] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:55:54.684443 31727 raft_consensus.cc:686] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee8eec859b940dead68aac2c0f5f8f4, State: Running, Role: LEADER
I20260809 00:55:54.684983 31727 consensus_queue.cc:237] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efab3b7a9a9942efafb119e28faa0642" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40699 } } peers { permanent_uuid: "1214beb866ba4414857f3476898ca05f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 44073 } } peers { permanent_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33633 } }
I20260809 00:55:54.690623 30916 catalog_manager.cc:5495] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee8eec859b940dead68aac2c0f5f8f4 (127.9.225.1). New cstate: current_term: 1 leader_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efab3b7a9a9942efafb119e28faa0642" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1214beb866ba4414857f3476898ca05f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 44073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee8eec859b940dead68aac2c0f5f8f4" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 33633 } health_report { overall_health: HEALTHY } } }
I20260809 00:55:54.720747 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.714463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.9.225.1:33633)
0809 00:55:54.714701 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:54.714707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:54.714746 (+ 39us) server_negotiation.cc:407] Connection header received
0809 00:55:54.714988 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:54.714992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:54.715087 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:54.715281 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:55:54.715710 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.716474 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.718017 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.718479 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.718626 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.719278 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:55:54.719902 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:55:54.720093 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:54.720157 (+ 64us) server_negotiation.cc:299] Negotiation successful
0809 00:55:54.720253 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 00:55:54.731362 31403 raft_consensus.cc:1261] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642 [term 1 FOLLOWER]: Refusing update from remote peer 5ee8eec859b940dead68aac2c0f5f8f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:55:54.731400 31610 raft_consensus.cc:1261] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f [term 1 FOLLOWER]: Refusing update from remote peer 5ee8eec859b940dead68aac2c0f5f8f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:55:54.732617 31727 consensus_queue.cc:1035] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efab3b7a9a9942efafb119e28faa0642" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:55:54.733013 31746 consensus_queue.cc:1035] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1214beb866ba4414857f3476898ca05f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 44073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:55:54.748024 31755 mvcc.cc:205] Tried to move back new op lower bound from 7316426566566129664 to 7316426566393913344. Current Snapshot: MvccSnapshot[applied={T|T < 7316426566566129664}]
I20260809 00:55:54.776396 30914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:55:54.776809 30914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:55:54.782678 30914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e31e32134c74e288f0398c883eefad2: Sending DeleteTablet for 3 replicas of tablet 1971aa9c4ded48abbac3eddd7e3a4103
I20260809 00:55:54.784713 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet 1971aa9c4ded48abbac3eddd7e3a4103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:55:54 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:51476
I20260809 00:55:54.784817 31371 tablet_service.cc:1514] Processing DeleteTablet for tablet 1971aa9c4ded48abbac3eddd7e3a4103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:55:54 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:33114
I20260809 00:55:54.788188 31122 tablet_service.cc:1514] Processing DeleteTablet for tablet 1971aa9c4ded48abbac3eddd7e3a4103 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:55:54 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:48076
I20260809 00:55:54.788861 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 30975
I20260809 00:55:54.803532 31772 tablet_replica.cc:330] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4: stopping tablet replica
I20260809 00:55:54.804623 31772 raft_consensus.cc:2229] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:55:54.806419 31772 raft_consensus.cc:2258] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:55:54.807319 31769 tablet_replica.cc:330] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f: stopping tablet replica
I20260809 00:55:54.808604 31769 raft_consensus.cc:2229] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:55:54.809311 31770 tablet_replica.cc:330] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642: stopping tablet replica
I20260809 00:55:54.809584 31769 raft_consensus.cc:2258] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:55:54.812055 31769 ts_tablet_manager.cc:1905] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:55:54.812069 31772 ts_tablet_manager.cc:1905] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:55:54.814409 31770 raft_consensus.cc:2229] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:55:54.815462 31770 raft_consensus.cc:2258] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:55:54.818809 31770 ts_tablet_manager.cc:1905] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:55:54.821954 31769 ts_tablet_manager.cc:1918] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:55:54.822347 31769 log.cc:1196] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/1971aa9c4ded48abbac3eddd7e3a4103
I20260809 00:55:54.823686 31769 ts_tablet_manager.cc:1939] T 1971aa9c4ded48abbac3eddd7e3a4103 P 1214beb866ba4414857f3476898ca05f: Deleting consensus metadata
I20260809 00:55:54.826577 30891 catalog_manager.cc:4841] TS 1214beb866ba4414857f3476898ca05f (127.9.225.3:44073): tablet 1971aa9c4ded48abbac3eddd7e3a4103 (table test-table [id=7e775bc0cd38470192eea85f4aa8bf1e]) successfully deleted
I20260809 00:55:54.829097 31770 ts_tablet_manager.cc:1918] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:55:54.829478 31770 log.cc:1196] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/1971aa9c4ded48abbac3eddd7e3a4103
I20260809 00:55:54.830250 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.793247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48092 (local address 127.9.225.1:33633)
0809 00:55:54.793562 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:54.793572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:54.802151 (+ 8579us) server_negotiation.cc:407] Connection header received
0809 00:55:54.802427 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:54.802437 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:54.802571 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:54.802749 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:54.806726 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.810266 (+ 3540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.812720 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.813834 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.813981 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.818673 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:54.818697 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:54.818718 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:54.818779 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:54.825828 (+ 7049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:54.827092 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:54.827106 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:54.827112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:54.827357 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:54.827892 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:54.827903 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:54.827907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:54.828451 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:54.828808 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:54.829566 (+ 758us) server_negotiation.cc:299] Negotiation successful
0809 00:55:54.829860 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260809 00:55:54.830389 31770 ts_tablet_manager.cc:1939] T 1971aa9c4ded48abbac3eddd7e3a4103 P efab3b7a9a9942efafb119e28faa0642: Deleting consensus metadata
I20260809 00:55:54.830708 31772 ts_tablet_manager.cc:1918] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:55:54.830976 31772 log.cc:1196] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/1971aa9c4ded48abbac3eddd7e3a4103
I20260809 00:55:54.831615 31772 ts_tablet_manager.cc:1939] T 1971aa9c4ded48abbac3eddd7e3a4103 P 5ee8eec859b940dead68aac2c0f5f8f4: Deleting consensus metadata
I20260809 00:55:54.832465 31228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:55:54.832844 30891 catalog_manager.cc:4841] TS efab3b7a9a9942efafb119e28faa0642 (127.9.225.2:40699): tablet 1971aa9c4ded48abbac3eddd7e3a4103 (table test-table [id=7e775bc0cd38470192eea85f4aa8bf1e]) successfully deleted
I20260809 00:55:54.935132 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 30975
I20260809 00:55:54.935169 30888 catalog_manager.cc:4841] TS 5ee8eec859b940dead68aac2c0f5f8f4 (127.9.225.1:33633): tablet 1971aa9c4ded48abbac3eddd7e3a4103 (table test-table [id=7e775bc0cd38470192eea85f4aa8bf1e]) successfully deleted
I20260809 00:55:54.961767 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 31240
I20260809 00:55:54.977706 31709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:54.962750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.9.225.2:40699)
0809 00:55:54.962974 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:54.962979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:54.962999 (+ 20us) server_negotiation.cc:407] Connection header received
0809 00:55:54.963104 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:54.963108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:54.963176 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:54.963278 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:54.964747 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.965628 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.967057 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:54.967516 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:54.967636 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:54.970662 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:54.970681 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:54.970701 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:54.970743 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:54.974090 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:54.975062 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:54.975071 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:54.975074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:54.975270 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:54.975882 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:54.975887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:54.975889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:54.976357 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:54.976647 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:54.977098 (+ 451us) server_negotiation.cc:299] Negotiation successful
0809 00:55:54.977313 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 00:55:54.978507 31467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:55:55.085026 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 31240
I20260809 00:55:55.126487 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 31483
I20260809 00:55:55.145850 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:55.127673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.9.225.3:44073)
0809 00:55:55.127912 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:55.127918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:55.127950 (+ 32us) server_negotiation.cc:407] Connection header received
0809 00:55:55.128045 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:55.128050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:55.128193 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:55.128325 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:55.130191 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:55.131608 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:55.133173 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:55.134211 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:55.134383 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:55.137813 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:55.137833 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:55.137862 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:55.137930 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:55.142395 (+ 4465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:55.143513 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:55.143524 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:55.143527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:55.143692 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:55.144108 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:55.144111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:55.144113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:55.144552 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:55.144828 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:55.145215 (+ 387us) server_negotiation.cc:299] Negotiation successful
0809 00:55:55.145530 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260809 00:55:55.146699 31677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:55:55.241496 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 31483
I20260809 00:55:55.274689 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 30859
I20260809 00:55:55.276008 30963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:55:55.395840 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 30859
I20260809 00:55:55.437572 10116 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 00:55:55.788946 10116 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 00:55:55.789175 10116 mini_ranger.cc:66] Stopping Ranger...
I20260809 00:55:56.146657 10116 mini_ranger.cc:68] Stopped Ranger
2026-08-09 00:55:56.147 UTC [29299] LOG: received smart shutdown request
2026-08-09 00:55:56.151 UTC [29299] LOG: background worker "logical replication launcher" (PID 29317) exited with exit code 1
2026-08-09 00:55:56.151 UTC [29304] LOG: shutting down
2026-08-09 00:55:56.316 UTC [29299] LOG: database system is shut down
2026-08-09T00:55:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (61069 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:55:56 dist-test-slave-j2vl krb5kdc[31794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:55:56 dist-test-slave-j2vl krb5kdc[31794](info): set up 2 sockets
Aug 09 00:55:56 dist-test-slave-j2vl krb5kdc[31794](info): commencing operation
krb5kdc: starting...
W20260809 00:55:57.860440 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.354s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:55:58 dist-test-slave-j2vl krb5kdc[31794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:55:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:55:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:55:58.189630 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35399
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:35399,127.9.225.61:40137,127.9.225.60:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:55:58.603582 31817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:55:58.603960 31817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:55:58.604130 31817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:55:58.612298 31817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:55:58.612434 31817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:55:58.612516 31817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:55:58.612607 31817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:55:58.612720 31817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:55:58.627919 31817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:35399,127.9.225.61:40137,127.9.225.60:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:55:58.629783 31817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:55:58.631366 31817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:55:58.645686 31825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:58.645920 31827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:58.645736 31829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:58.649290 31817 server_base.cc:1034] running on GCE node
I20260809 00:55:58.650494 31817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:55:58.652343 31817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:55:58.653635 31817 hybrid_clock.cc:648] HybridClock initialized: now 1786236958653573 us; error 60 us; skew 500 ppm
I20260809 00:55:58.654223 31817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:55:58 dist-test-slave-j2vl krb5kdc[31794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236958, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:58.658136 31817 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:55:58.660367 31817 webserver.cc:460] Webserver started at http://127.9.225.62:46683/ using document root <none> and password file <none>
I20260809 00:55:58.661285 31817 fs_manager.cc:359] Metadata directory not provided
I20260809 00:55:58.661486 31817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:55:58.661863 31817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:55:58.665602 31817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "7eab6a30aa63415d846f053ca4698c61"
format_stamp: "Formatted at 2026-08-09 00:55:58 on dist-test-slave-j2vl"
server_key: "7ac8209c9d2ff40484eabe1855844d3f"
server_key_iv: "0fcb8bfc8dba5f3df8d3dda777b0ddb9"
server_key_version: "encryptionkey@0"
I20260809 00:55:58.666734 31817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "7eab6a30aa63415d846f053ca4698c61"
format_stamp: "Formatted at 2026-08-09 00:55:58 on dist-test-slave-j2vl"
server_key: "7ac8209c9d2ff40484eabe1855844d3f"
server_key_iv: "0fcb8bfc8dba5f3df8d3dda777b0ddb9"
server_key_version: "encryptionkey@0"
I20260809 00:55:58.673216 31817 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260809 00:55:58.677780 31837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:58.679543 31817 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260809 00:55:58.680096 31817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "7eab6a30aa63415d846f053ca4698c61"
format_stamp: "Formatted at 2026-08-09 00:55:58 on dist-test-slave-j2vl"
server_key: "7ac8209c9d2ff40484eabe1855844d3f"
server_key_iv: "0fcb8bfc8dba5f3df8d3dda777b0ddb9"
server_key_version: "encryptionkey@0"
I20260809 00:55:58.680394 31817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:55:58.693902 31817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:55:58.700129 31817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:55:58.700548 31817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:55:58.723608 31817 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:35399
I20260809 00:55:58.723639 31912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:35399 every 8 connection(s)
I20260809 00:55:58.725425 31817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:55:58.730199 31913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:55:58.734885 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 31817
I20260809 00:55:58.735174 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:55:58.736069 10116 external_mini_cluster.cc:1428] Setting key 50e20ab6b705de2eaec094327fae6715
I20260809 00:55:58.741225 31913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:55:58 dist-test-slave-j2vl krb5kdc[31794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236958, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.225.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.61@KRBTEST.COM" created.
I20260809 00:55:58.776932 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:58.747102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35399 (local address 127.0.0.1:59672)
0809 00:55:58.749043 (+ 1941us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:58.749124 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:58.751338 (+ 2214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:58.752869 (+ 1531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:58.752884 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:58.754483 (+ 1599us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:55:58.755546 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:58.755598 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:58.757706 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:58.757724 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:58.760162 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:58.760192 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:58.762136 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:58.762147 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:58.762561 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:58.763754 (+ 1193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:55:58.763798 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:55:58.767309 (+ 3511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:55:58.771708 (+ 4399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:58.771722 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:58.771737 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:58.772257 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:58.772827 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:58.772832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:58.772835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:58.773022 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:58.774303 (+ 1281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:55:58.774315 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:55:58.774592 (+ 277us) client_negotiation.cc:769] Sending connection context
0809 00:55:58.774895 (+ 303us) client_negotiation.cc:240] Negotiation successful
0809 00:55:58.775400 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1599,"spinlock_wait_cycles":54656,"thread_start_us":1284,"threads_started":1}
I20260809 00:55:58.776948 31922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:58.747102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.9.225.62:35399)
0809 00:55:58.748726 (+ 1624us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:58.748762 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:58.749310 (+ 548us) server_negotiation.cc:407] Connection header received
0809 00:55:58.751872 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:58.751883 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:58.752310 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:58.752584 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:58.755954 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:58.757495 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:58.760428 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:58.761614 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:58.761883 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:58.767648 (+ 5765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:58.767659 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:58.767678 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:58.767714 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:58.771465 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:58.772472 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:58.772482 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:58.772484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:58.772683 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:58.773249 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:58.773255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:58.773259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:58.774086 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:58.774356 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:58.775095 (+ 739us) server_negotiation.cc:299] Negotiation successful
0809 00:55:58.775676 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":213,"server-negotiator.queue_time_us":1184,"thread_start_us":699,"threads_started":1}
I20260809 00:55:58.780570 31913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } has no permanent_uuid. Determining permanent_uuid...
I20260809 00:55:58.782004 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:58.781508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38062)
0809 00:55:58.781712 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:58.781851 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260809 00:55:58.782383 31842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
I20260809 00:55:58.784268 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:58.783816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38068)
0809 00:55:58.784018 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:58.784098 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260809 00:55:58.784514 31913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:40137: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.61.keytab.
Entry for principal kudu/127.9.225.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.61.keytab.
I20260809 00:55:58.818179 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.61:40137
--webserver_interface=127.9.225.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.61.keytab
--principal=kudu/127.9.225.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:35399,127.9.225.61:40137,127.9.225.60:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:55:58.834043 31913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } attempt: 1
I20260809 00:55:58.835583 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:58.834937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38080)
0809 00:55:58.835200 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:58.835357 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 00:55:58.836915 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:58.836388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38096)
0809 00:55:58.836600 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:58.836708 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 00:55:58.837175 31913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:40137: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
I20260809 00:55:58.905517 31913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } attempt: 2
I20260809 00:55:58.907047 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:58.906426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38098)
0809 00:55:58.906711 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:58.906844 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 00:55:58.908365 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:58.907872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38110)
0809 00:55:58.908055 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:58.908166 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260809 00:55:58.908612 31913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:40137: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
I20260809 00:55:58.999966 31913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } attempt: 3
I20260809 00:55:59.001431 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.000822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38122)
0809 00:55:59.001071 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.001189 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 00:55:59.002822 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.002299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38126)
0809 00:55:59.002488 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.002612 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 00:55:59.003254 31913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:40137: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
I20260809 00:55:59.146637 31913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } attempt: 4
I20260809 00:55:59.148092 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.147407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38130)
0809 00:55:59.147779 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.147908 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260809 00:55:59.149482 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.148940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38140)
0809 00:55:59.149138 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.149270 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 00:55:59.149958 31913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:40137: Network error: Client connection negotiation failed: client connection to 127.9.225.61:40137: connect: Connection refused (error 111)
W20260809 00:55:59.214874 31931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:55:59.215241 31931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:55:59.215445 31931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:55:59.223716 31931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:55:59.223872 31931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:55:59.223995 31931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:55:59.224092 31931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:55:59.224195 31931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:55:59.235554 31931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:35399,127.9.225.61:40137,127.9.225.60:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.61.keytab
--principal=kudu/127.9.225.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.61:40137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.225.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:55:59.237241 31931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:55:59.239019 31931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:55:59.252678 31948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:59.252620 31946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:59.252751 31944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:59.253791 31931 server_base.cc:1034] running on GCE node
I20260809 00:55:59.254815 31931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:55:59.256559 31931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:55:59.257843 31931 hybrid_clock.cc:648] HybridClock initialized: now 1786236959257781 us; error 53 us; skew 500 ppm
I20260809 00:55:59.258298 31931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:55:59 dist-test-slave-j2vl krb5kdc[31794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236959, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:59.262346 31931 init.cc:377] Logged in from keytab as kudu/127.9.225.61@KRBTEST.COM (short username kudu)
I20260809 00:55:59.264308 31931 webserver.cc:460] Webserver started at http://127.9.225.61:35369/ using document root <none> and password file <none>
I20260809 00:55:59.265163 31931 fs_manager.cc:359] Metadata directory not provided
I20260809 00:55:59.265332 31931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:55:59.265733 31931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:55:59.269209 31931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/data/instance:
uuid: "f870c66fffed4965a7fa548a9498a880"
format_stamp: "Formatted at 2026-08-09 00:55:59 on dist-test-slave-j2vl"
server_key: "9d762fee23b1f23d9070f3c5d309702d"
server_key_iv: "a8002bceba76c22e6872ab17f6603f85"
server_key_version: "encryptionkey@0"
I20260809 00:55:59.270304 31931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/wal/instance:
uuid: "f870c66fffed4965a7fa548a9498a880"
format_stamp: "Formatted at 2026-08-09 00:55:59 on dist-test-slave-j2vl"
server_key: "9d762fee23b1f23d9070f3c5d309702d"
server_key_iv: "a8002bceba76c22e6872ab17f6603f85"
server_key_version: "encryptionkey@0"
I20260809 00:55:59.276963 31931 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 00:55:59.281459 31958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:59.283102 31931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:55:59.283344 31931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/wal
uuid: "f870c66fffed4965a7fa548a9498a880"
format_stamp: "Formatted at 2026-08-09 00:55:59 on dist-test-slave-j2vl"
server_key: "9d762fee23b1f23d9070f3c5d309702d"
server_key_iv: "a8002bceba76c22e6872ab17f6603f85"
server_key_version: "encryptionkey@0"
I20260809 00:55:59.283587 31931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:55:59.316429 31931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:55:59.321521 31931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:55:59.321871 31931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:55:59.344157 31931 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.61:40137
I20260809 00:55:59.344267 32032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.61:40137 every 8 connection(s)
I20260809 00:55:59.345820 31931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
I20260809 00:55:59.350279 32033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:55:59.354705 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 31931
I20260809 00:55:59.354954 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-1/wal/instance
I20260809 00:55:59.355594 10116 external_mini_cluster.cc:1428] Setting key b75c05c4099bd817ba5ad9eff9235a07
I20260809 00:55:59.359885 32033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:55:59 dist-test-slave-j2vl krb5kdc[31794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236959, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.61@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.225.60@KRBTEST.COM; defaulting to no policy
I20260809 00:55:59.389438 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.364086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.9.225.62:35399)
0809 00:55:59.364669 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:59.364675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:59.366179 (+ 1504us) server_negotiation.cc:407] Connection header received
0809 00:55:59.369384 (+ 3205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:59.369394 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:59.369526 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:59.369670 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:59.371631 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:59.372917 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:59.374926 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:59.375732 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:59.375908 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:59.380147 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:59.380158 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:59.380166 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:59.380235 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:59.384713 (+ 4478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:59.386018 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:59.386028 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:59.386031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:59.386211 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:59.386873 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:59.386879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:59.386886 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:59.387362 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:59.387531 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:59.388665 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0809 00:55:59.388950 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":294,"threads_started":1}
I20260809 00:55:59.390849 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.364843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35399 (local address 127.0.0.1:59686)
0809 00:55:59.365921 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.365990 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:59.368930 (+ 2940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:59.369982 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:59.369992 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:59.370647 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:55:59.371366 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:59.371389 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:59.373164 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:59.373169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:59.374697 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:59.374736 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:59.376053 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:59.376060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:59.376318 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:59.377164 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:55:59.377192 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:55:59.379786 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:55:59.385011 (+ 5225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:59.385022 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:59.385041 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:59.385838 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:59.386370 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:59.386374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:59.386377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:59.386696 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:59.387537 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:55:59.387545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:55:59.388070 (+ 525us) client_negotiation.cc:769] Sending connection context
0809 00:55:59.388535 (+ 465us) client_negotiation.cc:240] Negotiation successful
0809 00:55:59.389274 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":443,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.60@KRBTEST.COM" created.
I20260809 00:55:59.393114 32033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:55:59 dist-test-slave-j2vl krb5kdc[31794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236959, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.61@KRBTEST.COM for kudu/127.9.225.61@KRBTEST.COM
I20260809 00:55:59.411149 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.394302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.9.225.61:40137)
0809 00:55:59.395063 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:59.395070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:59.395098 (+ 28us) server_negotiation.cc:407] Connection header received
0809 00:55:59.395148 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:59.395155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:59.395379 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:59.395560 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:59.396381 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:59.397598 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:59.398782 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:59.399407 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:59.399554 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:59.403531 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:59.403555 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:59.403581 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:59.403646 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:59.407347 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:59.408408 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:59.408417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:59.408419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:59.408573 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:59.409075 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:59.409079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:59.409082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:59.409661 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:59.409859 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:59.410421 (+ 562us) server_negotiation.cc:299] Negotiation successful
0809 00:55:59.410804 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":588,"thread_start_us":435,"threads_started":1}
I20260809 00:55:59.411271 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.393955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38146)
0809 00:55:59.394193 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.394213 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:59.394314 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:59.395543 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:59.395548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:59.396063 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:55:59.396201 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:59.396210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:59.397737 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:59.397744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:59.398626 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:59.398640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:59.399569 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:59.399574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:59.399881 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:59.400607 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:55:59.400627 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:55:59.403224 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:55:59.407544 (+ 4320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:59.407552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:59.407559 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:59.408200 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:59.408720 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:59.408728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:59.408734 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:59.408931 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:59.409809 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:55:59.409817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:55:59.410022 (+ 205us) client_negotiation.cc:769] Sending connection context
0809 00:55:59.410198 (+ 176us) client_negotiation.cc:240] Negotiation successful
0809 00:55:59.410639 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":20}
I20260809 00:55:59.414355 32033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } has no permanent_uuid. Determining permanent_uuid...
I20260809 00:55:59.415781 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.415208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43888)
0809 00:55:59.415436 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.415593 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260809 00:55:59.416136 31964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:55:59.417960 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.417570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43902)
0809 00:55:59.417733 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.417832 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 00:55:59.418167 32033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.60.keytab.
Entry for principal kudu/127.9.225.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.60.keytab.
I20260809 00:55:59.432960 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.60:45565
--webserver_interface=127.9.225.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.60.keytab
--principal=kudu/127.9.225.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:35399,127.9.225.61:40137,127.9.225.60:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:55:59.449271 31913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } attempt: 5
Aug 09 00:55:59 dist-test-slave-j2vl krb5kdc[31794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236958, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for kudu/127.9.225.61@KRBTEST.COM
I20260809 00:55:59.465965 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.450152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38154)
0809 00:55:59.450432 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.450473 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 00:55:59.450621 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:55:59.450964 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:55:59.450971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:55:59.451730 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:55:59.452028 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:59.452046 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:59.453142 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:59.453151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:59.454758 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:55:59.454778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:55:59.455578 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:55:59.455586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:55:59.455838 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:59.456560 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:55:59.456588 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:55:59.459581 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:55:59.463144 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:59.463151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:59.463155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:59.463603 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:59.464039 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:55:59.464044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:55:59.464048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:55:59.464207 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:55:59.464816 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:55:59.464822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:55:59.465020 (+ 198us) client_negotiation.cc:769] Sending connection context
0809 00:55:59.465183 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 00:55:59.465587 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 00:55:59.465965 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.450440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.9.225.61:40137)
0809 00:55:59.450624 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:59.450629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:59.450650 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:55:59.450742 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:59.450746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:59.450811 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:59.450891 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:55:59.452239 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:59.452983 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:59.454921 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:55:59.455400 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:55:59.455510 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:55:59.459856 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:55:59.459863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:55:59.459867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:55:59.459912 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:55:59.462942 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:59.463748 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:59.463754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:59.463758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:59.463911 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:55:59.464332 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:55:59.464338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:55:59.464343 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:55:59.464682 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:55:59.464858 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 00:55:59.465307 (+ 449us) server_negotiation.cc:299] Negotiation successful
0809 00:55:59.465607 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 00:55:59.467276 31913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } has no permanent_uuid. Determining permanent_uuid...
I20260809 00:55:59.467491 32033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 1
I20260809 00:55:59.468536 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.468073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43910)
0809 00:55:59.468270 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.468379 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 00:55:59.468611 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.468080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43912)
0809 00:55:59.468328 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.468435 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260809 00:55:59.469710 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.469319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43920)
0809 00:55:59.469531 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.469600 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260809 00:55:59.469723 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.469248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43916)
0809 00:55:59.469463 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.469558 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260809 00:55:59.469941 32033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
W20260809 00:55:59.470045 31913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:55:59.521370 31913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 1
I20260809 00:55:59.522898 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.522255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43932)
0809 00:55:59.522521 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.522648 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260809 00:55:59.524112 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.523686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43938)
0809 00:55:59.523862 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.523974 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 00:55:59.524379 31913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:55:59.538198 32033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 2
I20260809 00:55:59.539553 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.539036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43940)
0809 00:55:59.539258 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.539375 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260809 00:55:59.540771 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.540373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43950)
0809 00:55:59.540558 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.540639 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260809 00:55:59.541122 32033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:55:59.592669 31913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 2
I20260809 00:55:59.594233 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.593604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43958)
0809 00:55:59.593880 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.594012 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 00:55:59.595510 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.595085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43970)
0809 00:55:59.595272 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.595367 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 00:55:59.595765 31913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:55:59.632433 32033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 3
I20260809 00:55:59.634131 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.633569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43980)
0809 00:55:59.633835 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.633943 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 00:55:59.635452 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.634934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43984)
0809 00:55:59.635207 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.635303 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260809 00:55:59.635737 32033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:55:59.702078 31913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 3
I20260809 00:55:59.703639 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.703017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43986)
0809 00:55:59.703299 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.703419 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260809 00:55:59.704908 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.704405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:43990)
0809 00:55:59.704647 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.704740 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260809 00:55:59.705238 31913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:55:59.778997 32033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 4
I20260809 00:55:59.780632 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.779833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:44000)
0809 00:55:59.780161 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.780354 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 00:55:59.782030 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.781572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:44012)
0809 00:55:59.781767 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.781871 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 00:55:59.782332 32033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
W20260809 00:55:59.851051 32050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:55:59.851492 32050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:55:59.851663 32050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:55:59.859920 32050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:55:59.860122 32050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:55:59.860246 32050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:55:59.860354 32050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:55:59.860451 32050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:55:59.871675 32050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:35399,127.9.225.61:40137,127.9.225.60:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.60.keytab
--principal=kudu/127.9.225.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.60:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.225.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:55:59.873328 32050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:55:59.875025 32050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 00:55:59.882529 31913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 4
I20260809 00:55:59.883980 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.883300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:44028)
0809 00:55:59.883632 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.883762 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 00:55:59.885475 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.884824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:44036)
0809 00:55:59.885044 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 00:55:59.885263 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260809 00:55:59.885797 31913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
W20260809 00:55:59.891129 32062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:59.891153 32058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:55:59.891530 32056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:55:59.892238 32050 server_base.cc:1034] running on GCE node
I20260809 00:55:59.893214 32050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:55:59.895117 32050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:55:59.896404 32050 hybrid_clock.cc:648] HybridClock initialized: now 1786236959896303 us; error 100 us; skew 500 ppm
I20260809 00:55:59.896844 32050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:55:59 dist-test-slave-j2vl krb5kdc[31794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236959, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:55:59.900767 32050 init.cc:377] Logged in from keytab as kudu/127.9.225.60@KRBTEST.COM (short username kudu)
I20260809 00:55:59.903020 32050 webserver.cc:460] Webserver started at http://127.9.225.60:36463/ using document root <none> and password file <none>
I20260809 00:55:59.903941 32050 fs_manager.cc:359] Metadata directory not provided
I20260809 00:55:59.904124 32050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:55:59.904508 32050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:55:59.908149 32050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/data/instance:
uuid: "4b26819ace594e70b8753ca6ce6c32bf"
format_stamp: "Formatted at 2026-08-09 00:55:59 on dist-test-slave-j2vl"
server_key: "e156f75acae2cbb6a6d3e8d8b0ee24d5"
server_key_iv: "064258f6f62f1a2f8903ac0ae0c8e7dd"
server_key_version: "encryptionkey@0"
I20260809 00:55:59.909240 32050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/wal/instance:
uuid: "4b26819ace594e70b8753ca6ce6c32bf"
format_stamp: "Formatted at 2026-08-09 00:55:59 on dist-test-slave-j2vl"
server_key: "e156f75acae2cbb6a6d3e8d8b0ee24d5"
server_key_iv: "064258f6f62f1a2f8903ac0ae0c8e7dd"
server_key_version: "encryptionkey@0"
I20260809 00:55:59.915669 32050 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:55:59.920059 32073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:55:59.921839 32050 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:55:59.922101 32050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/wal
uuid: "4b26819ace594e70b8753ca6ce6c32bf"
format_stamp: "Formatted at 2026-08-09 00:55:59 on dist-test-slave-j2vl"
server_key: "e156f75acae2cbb6a6d3e8d8b0ee24d5"
server_key_iv: "064258f6f62f1a2f8903ac0ae0c8e7dd"
server_key_version: "encryptionkey@0"
I20260809 00:55:59.922371 32050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:55:59.951496 32050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:55:59.956535 32050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:55:59.956904 32050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:55:59.979285 32050 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.60:45565
I20260809 00:55:59.979334 32147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.60:45565 every 8 connection(s)
I20260809 00:55:59.981163 32050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
I20260809 00:55:59.986178 32148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:55:59.990687 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 32050
I20260809 00:55:59.990950 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786236719337175-10116-0/minicluster-data/master-2/wal/instance
I20260809 00:55:59.991485 10116 external_mini_cluster.cc:1428] Setting key cb7cdd70e0c8e19c8cf9c2f29ac40eff
I20260809 00:55:59.994684 32148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:56:00 dist-test-slave-j2vl krb5kdc[31794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.60@KRBTEST.COM
Aug 09 00:56:00 dist-test-slave-j2vl krb5kdc[31794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236959, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.60@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:00.021627 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.000179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.9.225.62:35399)
0809 00:56:00.000838 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:00.000847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:00.002207 (+ 1360us) server_negotiation.cc:407] Connection header received
0809 00:56:00.002493 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:00.002499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:00.002602 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:00.002730 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:00.004568 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.005678 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.007766 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.008208 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.008343 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.012801 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:00.012808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:00.012812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:00.012864 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:00.016680 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.018177 (+ 1497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.018184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.018188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.018479 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.019086 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.019092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.019095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.019567 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:00.019737 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:00.020556 (+ 819us) server_negotiation.cc:299] Negotiation successful
0809 00:56:00.020866 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":351,"threads_started":1}
I20260809 00:56:00.022441 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.000073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35399 (local address 127.0.0.1:59696)
0809 00:56:00.001987 (+ 1914us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:00.002046 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:00.002329 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:00.002827 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:00.002838 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:00.003921 (+ 1083us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:00.004352 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:00.004367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.005902 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.005906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:00.007522 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:00.007546 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.008523 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.008535 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:00.008782 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.009736 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:00.009765 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:00.012491 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:00.017116 (+ 4625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:00.017131 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:00.017147 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:00.017966 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:00.018664 (+ 698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:00.018669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:00.018673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:00.018948 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:00.019719 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:00.019728 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:00.020030 (+ 302us) client_negotiation.cc:769] Sending connection context
0809 00:56:00.020360 (+ 330us) client_negotiation.cc:240] Negotiation successful
0809 00:56:00.020964 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1608,"thread_start_us":1101,"threads_started":1}
I20260809 00:56:00.022503 32151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:55:59.993860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.9.225.60:45565)
0809 00:55:59.994823 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0809 00:55:59.994837 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:55:59.994891 (+ 54us) server_negotiation.cc:407] Connection header received
0809 00:55:59.996965 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:55:59.997001 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:55:59.997674 (+ 673us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:55:59.998137 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:00.000255 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.002924 (+ 2669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.005095 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.006263 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.006448 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.011994 (+ 5546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:00.012004 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:00.012022 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:00.012073 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:00.016738 (+ 4665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.018180 (+ 1442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.018191 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.018194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.018479 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.019086 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.019092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.019095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.019918 (+ 823us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:00.020155 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:00.021076 (+ 921us) server_negotiation.cc:299] Negotiation successful
0809 00:56:00.021418 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":451,"threads_started":1}
I20260809 00:56:00.024506 32148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:56:00 dist-test-slave-j2vl krb5kdc[31794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236959, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.60@KRBTEST.COM for kudu/127.9.225.61@KRBTEST.COM
Aug 09 00:56:00 dist-test-slave-j2vl krb5kdc[31794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:00.043057 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.025478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:40137 (local address 127.0.0.1:38170)
0809 00:56:00.025749 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:00.025770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:00.025892 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:00.026672 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:00.026677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:00.027314 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:00.027552 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:00.027567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.028988 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.028993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:00.030367 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:00.030392 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.031259 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.031268 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:00.031471 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.032195 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:00.032215 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:00.034955 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:00.039820 (+ 4865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:00.039829 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:00.039835 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:00.040340 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:00.040892 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:00.040896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:00.040899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:00.041097 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:00.041826 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:00.041831 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:00.042072 (+ 241us) client_negotiation.cc:769] Sending connection context
0809 00:56:00.042247 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 00:56:00.042717 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260809 00:56:00.043111 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.025569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.9.225.61:40137)
0809 00:56:00.026174 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:00.026179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:00.026197 (+ 18us) server_negotiation.cc:407] Connection header received
0809 00:56:00.026297 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:00.026301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:00.026392 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:00.026514 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:00.027927 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.028771 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.030607 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.031073 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.031190 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.035249 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:00.035261 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:00.035266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:00.035325 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:00.039600 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.040523 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.040529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.040532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.040722 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.041250 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.041256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.041259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.041691 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:00.041818 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:00.042436 (+ 618us) server_negotiation.cc:299] Negotiation successful
0809 00:56:00.042639 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":365,"threads_started":1}
I20260809 00:56:00.044190 32148 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } has no permanent_uuid. Determining permanent_uuid...
I20260809 00:56:00.045010 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.026488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.9.225.62:35399)
0809 00:56:00.026672 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:00.026679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:00.026697 (+ 18us) server_negotiation.cc:407] Connection header received
0809 00:56:00.026905 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:00.026910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:00.027012 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:00.027123 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:00.028689 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.029553 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.031313 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.031810 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.031979 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.037698 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:00.037705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:00.037709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:00.037778 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:00.041324 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.042393 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.042399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.042403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.042707 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.043263 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.043269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.043273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.043683 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:00.044032 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:00.044397 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 00:56:00.044671 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Aug 09 00:56:00 dist-test-slave-j2vl krb5kdc[31794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236959, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.60@KRBTEST.COM for kudu/127.9.225.60@KRBTEST.COM
Aug 09 00:56:00 dist-test-slave-j2vl krb5kdc[31794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.61@KRBTEST.COM
I20260809 00:56:00.061560 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.045034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:44052)
0809 00:56:00.045492 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:00.045513 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:00.045633 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:00.045980 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:00.045985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:00.046374 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:00.046576 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:00.046592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.047914 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.047919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:00.049417 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:00.049440 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.050799 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.050804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:00.051142 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.051845 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:00.051865 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:00.054754 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:00.058749 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:00.058756 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:00.058760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:00.059173 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:00.059658 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:00.059662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:00.059664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:00.059856 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:00.060597 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:00.060600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:00.060754 (+ 154us) client_negotiation.cc:769] Sending connection context
0809 00:56:00.060901 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 00:56:00.061173 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260809 00:56:00.061869 32151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.045422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.9.225.60:45565)
0809 00:56:00.045607 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:00.045613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:00.045631 (+ 18us) server_negotiation.cc:407] Connection header received
0809 00:56:00.045768 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:00.045771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:00.045829 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:00.045949 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:00.046759 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.047734 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.049682 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.050567 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.050729 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.054968 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:00.054973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:00.054976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:00.055009 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:00.058569 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.059313 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.059319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.059322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.059500 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.059995 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.060002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.060008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.060425 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:00.060618 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:00.061055 (+ 437us) server_negotiation.cc:299] Negotiation successful
0809 00:56:00.061410 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":75}
I20260809 00:56:00.066727 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.047724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.9.225.61:40137)
0809 00:56:00.047892 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:00.047902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:00.047919 (+ 17us) server_negotiation.cc:407] Connection header received
0809 00:56:00.048108 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:00.048111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:00.048175 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:00.048254 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:00.050244 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.051532 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.053109 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.053904 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.054063 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.059578 (+ 5515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:00.059589 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:00.059594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:00.059646 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:00.063277 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.064168 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.064175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.064179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.064354 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.064794 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.064800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.064804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.065140 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:00.065522 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:00.065877 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 00:56:00.066358 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 00:56:00.067728 32148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf: Bootstrap starting.
I20260809 00:56:00.072124 32148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:00.073035 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:56:00.073630 32148 log.cc:824] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:00.077011 32148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf: No bootstrap required, opened a new log
I20260809 00:56:00.081871 32033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 5
I20260809 00:56:00.097187 32148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:56:00.098049 32148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:00.098312 32148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b26819ace594e70b8753ca6ce6c32bf, State: Initialized, Role: FOLLOWER
I20260809 00:56:00.099599 32148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
Aug 09 00:56:00 dist-test-slave-j2vl krb5kdc[31794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236959, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.61@KRBTEST.COM for kudu/127.9.225.60@KRBTEST.COM
I20260809 00:56:00.105634 32151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.080510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.9.225.60:45565)
0809 00:56:00.080696 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:00.080702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:00.081389 (+ 687us) server_negotiation.cc:407] Connection header received
0809 00:56:00.081684 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:00.081690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:00.081777 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:00.081877 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:00.084773 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.086433 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.090157 (+ 3724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.091250 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.091388 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.096452 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:00.096592 (+ 140us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:00.096627 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:00.096743 (+ 116us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:00.101480 (+ 4737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.102486 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.102495 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.102501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.102694 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.103283 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.103290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.103295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.103854 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:00.104165 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:00.104890 (+ 725us) server_negotiation.cc:299] Negotiation successful
0809 00:56:00.105172 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 00:56:00.107100 32178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:56:00.107635 32178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:56:00.110041 32148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:56:00.110816 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.081601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.9.225.61:40137)
0809 00:56:00.081781 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:00.081787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:00.089911 (+ 8124us) server_negotiation.cc:407] Connection header received
0809 00:56:00.090185 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:00.090191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:00.090286 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:00.090361 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:00.093149 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.094341 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.096745 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.097429 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.097550 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.101583 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:00.101595 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:00.101604 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:00.101668 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:00.107005 (+ 5337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.108022 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.108028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.108032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.108199 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.108702 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.108709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.108712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.109134 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:00.109483 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:00.110124 (+ 641us) server_negotiation.cc:299] Negotiation successful
0809 00:56:00.110424 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 00:56:00.116297 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.082562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:44056)
0809 00:56:00.082966 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:00.082990 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:00.083112 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:00.089065 (+ 5953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:00.089071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:00.089747 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:00.090024 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:00.090044 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.094143 (+ 4099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.094152 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:00.095802 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:00.095830 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.099991 (+ 4161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.100000 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:00.100296 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.101023 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:00.101052 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:00.105148 (+ 4096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:00.112678 (+ 7530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:00.112688 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:00.112694 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:00.113326 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:00.113971 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:00.113980 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:00.113987 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:00.114230 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:00.115134 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:00.115142 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:00.115341 (+ 199us) client_negotiation.cc:769] Sending connection context
0809 00:56:00.115497 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 00:56:00.115849 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269}
I20260809 00:56:00.117393 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.079898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.9.225.62:35399)
0809 00:56:00.080162 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:00.080172 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:00.081018 (+ 846us) server_negotiation.cc:407] Connection header received
0809 00:56:00.081485 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:00.081492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:00.081599 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:00.081684 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:00.085164 (+ 3480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.086746 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.090978 (+ 4232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.091709 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.091840 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.109035 (+ 17195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:00.109042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:00.109045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:00.109097 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:00.113642 (+ 4545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.114605 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.114614 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.114620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.114832 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.115370 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.115376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.115380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.115782 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:00.116112 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:00.116612 (+ 500us) server_negotiation.cc:299] Negotiation successful
0809 00:56:00.116931 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 00:56:00.117899 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.083496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.9.225.60:45565)
0809 00:56:00.087472 (+ 3976us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:00.087478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:00.087512 (+ 34us) server_negotiation.cc:407] Connection header received
0809 00:56:00.087581 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:00.087587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:00.087698 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:00.090454 (+ 2756us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:00.090788 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.093154 (+ 2366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.098260 (+ 5106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.099702 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.105370 (+ 5668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.105642 (+ 272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:00.105651 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:00.105659 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:00.107937 (+ 2278us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:00.112487 (+ 4550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.113576 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.113582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.113585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.113794 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.114419 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.114425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.114429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.114947 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:00.116969 (+ 2022us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:00.117151 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 00:56:00.117469 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3804,"thread_start_us":3591,"threads_started":1}
I20260809 00:56:00.124213 32033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880: Bootstrap starting.
W20260809 00:56:00.127063 32190 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 00:56:00.127363 32190 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 00:56:00.129590 32033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:00.131235 32033 log.cc:824] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:00.135275 32033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880: No bootstrap required, opened a new log
I20260809 00:56:00.123907 32050 master_runner.cc:186] Error getting master registration for 127.9.225.62:35399: OK, instance_id { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" instance_seqno: 1786236958716415 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 00:56:00.137281 32050 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 00:56:00.144582 32033 raft_consensus.cc:348] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:56:00.144989 32033 raft_consensus.cc:374] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:00.145152 32033 raft_consensus.cc:729] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f870c66fffed4965a7fa548a9498a880, State: Initialized, Role: FOLLOWER
I20260809 00:56:00.145918 32033 consensus_queue.cc:260] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:56:00.148998 32199 sys_catalog.cc:455] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:56:00.149511 32199 sys_catalog.cc:458] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:56:00.150523 32033 sys_catalog.cc:564] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:56:00.163080 31913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 5
I20260809 00:56:00.159664 31931 master_runner.cc:186] Error getting master registration for 127.9.225.62:35399: OK, instance_id { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" instance_seqno: 1786236958716415 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 00:56:00.170435 31931 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 00:56:00.174432 32219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 00:56:00.174685 32219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 09 00:56:00 dist-test-slave-j2vl krb5kdc[31794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236958, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for kudu/127.9.225.60@KRBTEST.COM
I20260809 00:56:00.185770 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.163886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:44060)
0809 00:56:00.164315 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:00.164337 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:00.164462 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:00.165467 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:00.165472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:00.166052 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:00.166283 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:00.166296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.167870 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.167876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:00.169452 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:00.169471 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.172808 (+ 3337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.172816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:00.173025 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.173791 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:00.173817 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:00.177224 (+ 3407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:00.182222 (+ 4998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:00.182229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:00.182233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:00.182864 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:00.183453 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:00.183459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:00.183462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:00.183675 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:00.184530 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:00.184536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:00.184724 (+ 188us) client_negotiation.cc:769] Sending connection context
0809 00:56:00.184897 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 00:56:00.185239 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271}
I20260809 00:56:00.186997 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:00.164695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.9.225.60:45565)
0809 00:56:00.164900 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:00.164906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:00.164925 (+ 19us) server_negotiation.cc:407] Connection header received
0809 00:56:00.164991 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:00.164997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:00.165101 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:00.165204 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:00.166477 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.167678 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.169647 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:00.170387 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:00.170553 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:00.177558 (+ 7005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:00.177566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:00.177572 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:00.177627 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:00.181954 (+ 4327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.183061 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.183067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.183073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.183283 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:00.183828 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:00.183834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:00.183837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:00.184251 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:00.184451 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:00.186319 (+ 1868us) server_negotiation.cc:299] Negotiation successful
0809 00:56:00.186620 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 00:56:00.194250 31913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61: Bootstrap starting.
I20260809 00:56:00.199520 31913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:00.201243 31913 log.cc:824] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:00.207746 31913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61: No bootstrap required, opened a new log
W20260809 00:56:00.210142 32079 tablet.cc:2369] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:56:00.220777 31913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:56:00.221467 31913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:00.221642 31913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eab6a30aa63415d846f053ca4698c61, State: Initialized, Role: FOLLOWER
I20260809 00:56:00.222637 32178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:56:00.222725 31913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:56:00.223017 32178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:56:00.225361 32178 leader_election.cc:288] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eab6a30aa63415d846f053ca4698c61 (127.9.225.62:35399), f870c66fffed4965a7fa548a9498a880 (127.9.225.61:40137)
I20260809 00:56:00.228099 31995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b26819ace594e70b8753ca6ce6c32bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f870c66fffed4965a7fa548a9498a880" is_pre_election: true
I20260809 00:56:00.229054 31995 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b26819ace594e70b8753ca6ce6c32bf in term 0.
I20260809 00:56:00.230558 32077 leader_election.cc:302] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b26819ace594e70b8753ca6ce6c32bf, f870c66fffed4965a7fa548a9498a880; no voters:
I20260809 00:56:00.231475 32178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:56:00.231818 32178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:56:00.232095 32178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:00.242112 32178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:56:00.249629 31995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b26819ace594e70b8753ca6ce6c32bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f870c66fffed4965a7fa548a9498a880"
I20260809 00:56:00.250080 31995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:00.271157 32178 leader_election.cc:288] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [CANDIDATE]: Term 1 election: Requested vote from peers 7eab6a30aa63415d846f053ca4698c61 (127.9.225.62:35399), f870c66fffed4965a7fa548a9498a880 (127.9.225.61:40137)
I20260809 00:56:00.273007 31995 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b26819ace594e70b8753ca6ce6c32bf in term 1.
I20260809 00:56:00.277331 32077 leader_election.cc:302] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b26819ace594e70b8753ca6ce6c32bf, f870c66fffed4965a7fa548a9498a880; no voters:
I20260809 00:56:00.278160 32178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:56:00.282202 32222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:56:00.283281 31913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:56:00.283324 31875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b26819ace594e70b8753ca6ce6c32bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eab6a30aa63415d846f053ca4698c61"
I20260809 00:56:00.283682 31876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b26819ace594e70b8753ca6ce6c32bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eab6a30aa63415d846f053ca4698c61" is_pre_election: true
I20260809 00:56:00.284057 32222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:56:00.284147 31875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:00.296178 31875 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b26819ace594e70b8753ca6ce6c32bf in term 1.
I20260809 00:56:00.297516 32178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [term 1 LEADER]: Becoming Leader. State: Replica: 4b26819ace594e70b8753ca6ce6c32bf, State: Running, Role: LEADER
I20260809 00:56:00.298480 32178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:56:00.322661 32224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b26819ace594e70b8753ca6ce6c32bf. Latest consensus state: current_term: 1 leader_uuid: "4b26819ace594e70b8753ca6ce6c32bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:56:00.323299 32224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [sys.catalog]: This master's current role is: LEADER
W20260809 00:56:00.325094 31968 tablet.cc:2369] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:56:00.334100 32237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:56:00.337005 32237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:56:00.368204 31995 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [term 1 FOLLOWER]: Refusing update from remote peer 4b26819ace594e70b8753ca6ce6c32bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:56:00.369508 32224 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:56:00.377816 31875 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [term 1 FOLLOWER]: Refusing update from remote peer 4b26819ace594e70b8753ca6ce6c32bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:56:00.379068 32224 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:56:00.413829 32222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b26819ace594e70b8753ca6ce6c32bf. Latest consensus state: current_term: 1 leader_uuid: "4b26819ace594e70b8753ca6ce6c32bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:56:00.414256 32222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:56:00.422899 32242 mvcc.cc:205] Tried to move back new op lower bound from 7316426589652430848 to 7316426589393444864. Current Snapshot: MvccSnapshot[applied={T|T < 7316426589652430848}]
I20260809 00:56:00.427668 32250 mvcc.cc:205] Tried to move back new op lower bound from 7316426589652430848 to 7316426589393444864. Current Snapshot: MvccSnapshot[applied={T|T < 7316426589652430848}]
I20260809 00:56:00.430962 32199 sys_catalog.cc:455] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b26819ace594e70b8753ca6ce6c32bf. Latest consensus state: current_term: 1 leader_uuid: "4b26819ace594e70b8753ca6ce6c32bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:56:00.431401 32199 sys_catalog.cc:458] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:56:00.439488 32222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b26819ace594e70b8753ca6ce6c32bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:56:00.439888 32222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:56:00.440351 32247 mvcc.cc:205] Tried to move back new op lower bound from 7316426589652430848 to 7316426589393444864. Current Snapshot: MvccSnapshot[applied={T|T < 7316426589652430848}]
I20260809 00:56:00.449106 32199 sys_catalog.cc:455] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b26819ace594e70b8753ca6ce6c32bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:56:00.449445 32199 sys_catalog.cc:458] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:56:00.453718 32224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b26819ace594e70b8753ca6ce6c32bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:56:00.454078 32224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:00.454401 32224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b26819ace594e70b8753ca6ce6c32bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eab6a30aa63415d846f053ca4698c61" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35399 } } peers { permanent_uuid: "f870c66fffed4965a7fa548a9498a880" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 40137 } } peers { permanent_uuid: "4b26819ace594e70b8753ca6ce6c32bf" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:56:00.454630 32224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:00.458384 32237 catalog_manager.cc:1347] Generated new cluster ID: dd1ee4fec56746b8b79d53ca604ecad6
I20260809 00:56:00.458527 32237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:56:00.512056 32258 catalog_manager.cc:1259] Loaded cluster ID: dd1ee4fec56746b8b79d53ca604ecad6
I20260809 00:56:00.512189 32258 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61: loading cluster ID for follower catalog manager: success
W20260809 00:56:00.513643 32258 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260809 00:56:00.513819 32258 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260809 00:56:00.536984 32237 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:56:00.539382 32237 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:56:00.567770 32237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b26819ace594e70b8753ca6ce6c32bf: Generated new TSK 0
I20260809 00:56:00.569110 32237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:56:01.177181 32219 catalog_manager.cc:1259] Loaded cluster ID: dd1ee4fec56746b8b79d53ca604ecad6
I20260809 00:56:01.177304 32219 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880: loading cluster ID for follower catalog manager: success
I20260809 00:56:01.180347 32219 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880: acquiring CA information for follower catalog manager: success
I20260809 00:56:01.182523 32219 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f870c66fffed4965a7fa548a9498a880: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 00:56:01.517534 32258 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61: acquiring CA information for follower catalog manager: success
I20260809 00:56:01.519927 32258 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7eab6a30aa63415d846f053ca4698c61: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 00:56:01.751048 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 31817
I20260809 00:56:01.752432 31901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:01.867705 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 31817
I20260809 00:56:01.901144 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 31931
I20260809 00:56:01.902436 32026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:02.024068 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 31931
I20260809 00:56:02.058624 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 32050
I20260809 00:56:02.059895 32141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:02.181941 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 32050
2026-08-09T00:56:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5732 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:56:02 dist-test-slave-j2vl krb5kdc[32275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:56:02 dist-test-slave-j2vl krb5kdc[32275](info): set up 2 sockets
Aug 09 00:56:02 dist-test-slave-j2vl krb5kdc[32275](info): commencing operation
krb5kdc: starting...
W20260809 00:56:03.568727 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.328s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:56:03 dist-test-slave-j2vl krb5kdc[32275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:56:03.917835 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39007
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:39007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:56:04.303597 32297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:04.304028 32297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:04.304210 32297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:04.311872 32297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:56:04.312071 32297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:04.312193 32297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:04.312306 32297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:56:04.312407 32297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:56:04.322970 32297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:39007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:04.324610 32297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:04.326110 32297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:04.339422 32305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:04.339359 32309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:04.339401 32307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:04.340291 32297 server_base.cc:1034] running on GCE node
I20260809 00:56:04.341231 32297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:04.342968 32297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:04.344280 32297 hybrid_clock.cc:648] HybridClock initialized: now 1786236964344212 us; error 47 us; skew 500 ppm
I20260809 00:56:04.344762 32297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:04 dist-test-slave-j2vl krb5kdc[32275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236964, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:04.348742 32297 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:56:04.350937 32297 webserver.cc:460] Webserver started at http://127.9.225.62:35249/ using document root <none> and password file <none>
I20260809 00:56:04.351816 32297 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:04.352015 32297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:04.352389 32297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:56:04.356086 32297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "050fae0a59894ebd821b402d678ad772"
format_stamp: "Formatted at 2026-08-09 00:56:04 on dist-test-slave-j2vl"
server_key: "1c4a56f9e92951c532f38cd00a6bf9e0"
server_key_iv: "9e92a8fa26722352a99b232d9f67c594"
server_key_version: "encryptionkey@0"
I20260809 00:56:04.357093 32297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "050fae0a59894ebd821b402d678ad772"
format_stamp: "Formatted at 2026-08-09 00:56:04 on dist-test-slave-j2vl"
server_key: "1c4a56f9e92951c532f38cd00a6bf9e0"
server_key_iv: "9e92a8fa26722352a99b232d9f67c594"
server_key_version: "encryptionkey@0"
I20260809 00:56:04.363819 32297 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:56:04.368305 32317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:04.370249 32297 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:56:04.370508 32297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "050fae0a59894ebd821b402d678ad772"
format_stamp: "Formatted at 2026-08-09 00:56:04 on dist-test-slave-j2vl"
server_key: "1c4a56f9e92951c532f38cd00a6bf9e0"
server_key_iv: "9e92a8fa26722352a99b232d9f67c594"
server_key_version: "encryptionkey@0"
I20260809 00:56:04.370849 32297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:04.393980 32297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:04.399185 32297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:04.399525 32297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:04.421100 32297 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:39007
I20260809 00:56:04.421131 32407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:39007 every 8 connection(s)
I20260809 00:56:04.422814 32297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:56:04.427192 32408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:04.432541 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 32297
I20260809 00:56:04.432826 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:56:04.433811 10116 external_mini_cluster.cc:1428] Setting key 36607cd3c3037bef18d9a6fa2041d3ca
I20260809 00:56:04.441164 32408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772: Bootstrap starting.
I20260809 00:56:04.445453 32408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:04.446835 32408 log.cc:824] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:04.450922 32408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772: No bootstrap required, opened a new log
Aug 09 00:56:04 dist-test-slave-j2vl krb5kdc[32275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:04.461128 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:04.436078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.9.225.62:39007)
0809 00:56:04.437154 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:04.437167 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:04.437223 (+ 56us) server_negotiation.cc:407] Connection header received
0809 00:56:04.439269 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:04.439297 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:04.439766 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:04.440159 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:04.442305 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:04.443698 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:04.445748 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:04.446537 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:04.446692 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:04.452376 (+ 5684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:04.452386 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:04.452407 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:04.452453 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:04.456118 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:04.457085 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:04.457093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:04.457102 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:04.457312 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:04.457801 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:04.457806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:04.457808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:04.458505 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:04.458906 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:04.459453 (+ 547us) server_negotiation.cc:299] Negotiation successful
0809 00:56:04.459986 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":490,"threads_started":1}
I20260809 00:56:04.461109 32408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050fae0a59894ebd821b402d678ad772" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39007 } }
I20260809 00:56:04.461704 32408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:04.461848 32408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050fae0a59894ebd821b402d678ad772, State: Initialized, Role: FOLLOWER
I20260809 00:56:04.462683 32408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050fae0a59894ebd821b402d678ad772" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39007 } }
I20260809 00:56:04.463069 32408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:56:04.463227 32408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:56:04.463502 32408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:04.467425 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:56:04.467644 32408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050fae0a59894ebd821b402d678ad772" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39007 } }
I20260809 00:56:04.468438 32408 leader_election.cc:302] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 050fae0a59894ebd821b402d678ad772; no voters:
I20260809 00:56:04.469265 32408 leader_election.cc:288] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:56:04.469408 32413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:56:04.471155 32413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [term 1 LEADER]: Becoming Leader. State: Replica: 050fae0a59894ebd821b402d678ad772, State: Running, Role: LEADER
I20260809 00:56:04.471807 32413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050fae0a59894ebd821b402d678ad772" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39007 } }
I20260809 00:56:04.472270 32408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:56:04.475690 32415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 050fae0a59894ebd821b402d678ad772. Latest consensus state: current_term: 1 leader_uuid: "050fae0a59894ebd821b402d678ad772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050fae0a59894ebd821b402d678ad772" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39007 } } }
I20260809 00:56:04.476015 32415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:04.475140 32420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "050fae0a59894ebd821b402d678ad772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050fae0a59894ebd821b402d678ad772" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39007 } } }
I20260809 00:56:04.476617 32420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772 [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:04.478102 32424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:56:04.484493 32424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:56:04.495626 32424 catalog_manager.cc:1347] Generated new cluster ID: 6d8dddba9fbb43e8aafc3fb7307589a9
I20260809 00:56:04.495784 32424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:56:04.512009 32424 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:56:04.513644 32424 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:56:04.522397 32424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 050fae0a59894ebd821b402d678ad772: Generated new TSK 0
I20260809 00:56:04.523617 32424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:56:04.582837 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 32297
I20260809 00:56:04.584407 32400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:04.688620 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 32297
2026-08-09T00:56:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2498 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:56:04 dist-test-slave-j2vl krb5kdc[32452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:56:04 dist-test-slave-j2vl krb5kdc[32452](info): set up 2 sockets
Aug 09 00:56:04 dist-test-slave-j2vl krb5kdc[32452](info): commencing operation
krb5kdc: starting...
W20260809 00:56:06.082849 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.344s user 0.003s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:56:06 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:56:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:56:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:56:06.413267 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35871
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:35871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:56:06.801004 32478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:06.801397 32478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:06.801496 32478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:06.809080 32478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:56:06.809226 32478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:06.809275 32478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:06.809327 32478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:56:06.809506 32478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:56:06.820308 32478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:35871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:06.821950 32478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:06.823417 32478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:06.836961 32486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:06.836963 32490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:06.836910 32487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:06.843590 32478 server_base.cc:1034] running on GCE node
I20260809 00:56:06.844794 32478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:06.846589 32478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:06.847836 32478 hybrid_clock.cc:648] HybridClock initialized: now 1786236966847762 us; error 62 us; skew 500 ppm
I20260809 00:56:06.848304 32478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:06 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236966, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:06.855784 32478 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:56:06.857842 32478 webserver.cc:460] Webserver started at http://127.9.225.62:43677/ using document root <none> and password file <none>
I20260809 00:56:06.858722 32478 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:06.858909 32478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:06.859282 32478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:56:06.862886 32478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "25d7d397d6734f85a2f483ee67352746"
format_stamp: "Formatted at 2026-08-09 00:56:06 on dist-test-slave-j2vl"
server_key: "2ddd14e9279df414455ac81771e4af87"
server_key_iv: "0f81b8ba5af581ba62706611f8a53739"
server_key_version: "encryptionkey@0"
I20260809 00:56:06.863791 32478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "25d7d397d6734f85a2f483ee67352746"
format_stamp: "Formatted at 2026-08-09 00:56:06 on dist-test-slave-j2vl"
server_key: "2ddd14e9279df414455ac81771e4af87"
server_key_iv: "0f81b8ba5af581ba62706611f8a53739"
server_key_version: "encryptionkey@0"
I20260809 00:56:06.870128 32478 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 00:56:06.874294 32497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:06.875921 32478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260809 00:56:06.876157 32478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "25d7d397d6734f85a2f483ee67352746"
format_stamp: "Formatted at 2026-08-09 00:56:06 on dist-test-slave-j2vl"
server_key: "2ddd14e9279df414455ac81771e4af87"
server_key_iv: "0f81b8ba5af581ba62706611f8a53739"
server_key_version: "encryptionkey@0"
I20260809 00:56:06.876420 32478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:06.892212 32478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:06.897332 32478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:06.897848 32478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:06.919209 32478 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:35871
I20260809 00:56:06.919230 32582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:35871 every 8 connection(s)
I20260809 00:56:06.920871 32478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:56:06.925806 32584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:06.927350 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 32478
I20260809 00:56:06.927608 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:56:06.928539 10116 external_mini_cluster.cc:1428] Setting key 07f73ec30db7de3e6f70e23d5bce85ad
I20260809 00:56:06.942838 32584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Bootstrap starting.
Aug 09 00:56:06 dist-test-slave-j2vl krb5kdc[32452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:06.947430 32584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:06.949084 32584 log.cc:824] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:06.953266 32584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: No bootstrap required, opened a new log
I20260809 00:56:06.956918 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:06.931078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.9.225.62:35871)
0809 00:56:06.932047 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:06.932061 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:06.932108 (+ 47us) server_negotiation.cc:407] Connection header received
0809 00:56:06.933932 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:06.933964 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:06.934460 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:06.934856 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:06.936907 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:06.938420 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:06.940415 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:06.941165 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:06.941328 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:06.946863 (+ 5535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:06.946880 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:06.946906 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:06.946974 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:06.951025 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:06.952193 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:06.952205 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:06.952215 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:06.952489 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:06.952982 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:06.952988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:06.952991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:06.953892 (+ 901us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:06.954277 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:06.954758 (+ 481us) server_negotiation.cc:299] Negotiation successful
0809 00:56:06.955301 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":463,"threads_started":1}
I20260809 00:56:06.963227 32584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } }
I20260809 00:56:06.963891 32584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:06.963995 32584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d7d397d6734f85a2f483ee67352746, State: Initialized, Role: FOLLOWER
I20260809 00:56:06.964609 32584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } }
I20260809 00:56:06.964919 32584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:56:06.965035 32584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:56:06.965178 32584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:06.968119 32584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } }
I20260809 00:56:06.968677 32584 leader_election.cc:302] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25d7d397d6734f85a2f483ee67352746; no voters:
I20260809 00:56:06.969275 32584 leader_election.cc:288] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:56:06.969504 32590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:56:06.970307 32590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 1 LEADER]: Becoming Leader. State: Replica: 25d7d397d6734f85a2f483ee67352746, State: Running, Role: LEADER
I20260809 00:56:06.971014 32590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } }
I20260809 00:56:06.972028 32584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:56:06.974375 32591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25d7d397d6734f85a2f483ee67352746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } } }
I20260809 00:56:06.974891 32591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:06.975340 32592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25d7d397d6734f85a2f483ee67352746. Latest consensus state: current_term: 1 leader_uuid: "25d7d397d6734f85a2f483ee67352746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } } }
I20260809 00:56:06.975754 32592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:06.977978 32598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:56:06.984335 32598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:56:06.995380 32598 catalog_manager.cc:1347] Generated new cluster ID: ee8a326af34041dda221f75293b6c567
I20260809 00:56:06.995545 32598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:56:07.005211 32598 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:56:07.006803 32598 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:56:07.019759 32598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Generated new TSK 0
I20260809 00:56:07.020892 32598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:56:07.109105 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35871
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:56:07.498306 32622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:07.498663 32622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:07.498828 32622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:07.506791 32622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:07.506935 32622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:07.507063 32622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:56:07.518149 32622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:07.519635 32622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:07.521178 32622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:07.535992 32630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:07.536063 32635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:07.536063 32629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:07.537066 32622 server_base.cc:1034] running on GCE node
I20260809 00:56:07.537883 32622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:07.539000 32622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:07.540266 32622 hybrid_clock.cc:648] HybridClock initialized: now 1786236967540192 us; error 61 us; skew 500 ppm
I20260809 00:56:07.540637 32622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:07 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236967, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:07.544224 32622 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:56:07.545974 32622 webserver.cc:460] Webserver started at http://127.9.225.1:44941/ using document root <none> and password file <none>
I20260809 00:56:07.546769 32622 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:07.546931 32622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:07.547266 32622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:56:07.550653 32622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "156cb7dab3294f11a63a1cb8e16f5afe"
format_stamp: "Formatted at 2026-08-09 00:56:07 on dist-test-slave-j2vl"
server_key: "0dcbe8098e776157e256bf5babda35e6"
server_key_iv: "d78ac24cb8d98107e95f64bad47f43b4"
server_key_version: "encryptionkey@0"
I20260809 00:56:07.551581 32622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "156cb7dab3294f11a63a1cb8e16f5afe"
format_stamp: "Formatted at 2026-08-09 00:56:07 on dist-test-slave-j2vl"
server_key: "0dcbe8098e776157e256bf5babda35e6"
server_key_iv: "d78ac24cb8d98107e95f64bad47f43b4"
server_key_version: "encryptionkey@0"
I20260809 00:56:07.557871 32622 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 00:56:07.562075 32642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:07.563748 32622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 00:56:07.564004 32622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "156cb7dab3294f11a63a1cb8e16f5afe"
format_stamp: "Formatted at 2026-08-09 00:56:07 on dist-test-slave-j2vl"
server_key: "0dcbe8098e776157e256bf5babda35e6"
server_key_iv: "d78ac24cb8d98107e95f64bad47f43b4"
server_key_version: "encryptionkey@0"
I20260809 00:56:07.564286 32622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:07.589033 32622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:07.594429 32622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:07.594798 32622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:07.596001 32622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:07.597625 32622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:07.597711 32622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:07.597808 32622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:07.597855 32622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:07.633306 32622 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:38923
I20260809 00:56:07.633392 356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:38923 every 8 connection(s)
I20260809 00:56:07.634884 32622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:56:07.637099 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 32622
I20260809 00:56:07.637389 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:56:07.638193 10116 external_mini_cluster.cc:1428] Setting key 27e1c223a45d4b7dc87c957181f01fcc
Aug 09 00:56:07 dist-test-slave-j2vl krb5kdc[32452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236967, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:07.657994 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:07.638162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:49835 (local address 127.9.225.62:35871)
0809 00:56:07.638717 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:07.638724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:07.639816 (+ 1092us) server_negotiation.cc:407] Connection header received
0809 00:56:07.641910 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:07.641917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:07.642015 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:07.642143 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:07.643900 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:07.644641 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:07.646386 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:07.646812 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:07.646978 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:07.650926 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:07.650932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:07.650936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:07.650973 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:07.653995 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:07.655184 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:07.655189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:07.655192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:07.655343 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:07.655906 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:07.655910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:07.655912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:07.656205 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:07.656324 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:07.657496 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0809 00:56:07.657714 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":278,"threads_started":1}
I20260809 00:56:07.659734 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:07.638763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:49835)
0809 00:56:07.639645 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:07.639689 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:07.641591 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:07.642431 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:07.642441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:07.643038 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:07.643683 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:07.643706 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:07.644831 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:07.644835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:07.646215 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:07.646229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:07.647086 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:07.647095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:07.647358 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:07.648141 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:07.648167 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:07.650628 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:07.654188 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:07.654201 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:07.654226 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:07.655003 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:07.655480 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:07.655488 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:07.655494 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:07.655779 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:07.656342 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:07.656354 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:07.656897 (+ 543us) client_negotiation.cc:769] Sending connection context
0809 00:56:07.657312 (+ 415us) client_negotiation.cc:240] Negotiation successful
0809 00:56:07.658049 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":357,"threads_started":1}
I20260809 00:56:07.661700 357 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35871
I20260809 00:56:07.662107 357 heartbeater.cc:464] Registering TS with master...
I20260809 00:56:07.663183 357 heartbeater.cc:511] Master 127.9.225.62:35871 requested a full tablet report, sending...
I20260809 00:56:07.666491 32528 ts_manager.cc:194] Registered new tserver with Master: 156cb7dab3294f11a63a1cb8e16f5afe (127.9.225.1:38923)
I20260809 00:56:07.669032 32528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:49835
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:56:07.718624 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35871
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:56:08.096697 371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:08.097143 371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:08.097359 371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:08.105741 371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:08.105969 371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:08.106184 371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:56:08.117632 371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:08.119287 371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:08.120906 371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:08.136135 377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:08.136166 379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:08.136102 376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:08.137665 371 server_base.cc:1034] running on GCE node
I20260809 00:56:08.138511 371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:08.139787 371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:08.141081 371 hybrid_clock.cc:648] HybridClock initialized: now 1786236968141005 us; error 59 us; skew 500 ppm
I20260809 00:56:08.141537 371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:08 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236968, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:08.145246 371 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:56:08.147100 371 webserver.cc:460] Webserver started at http://127.9.225.2:41007/ using document root <none> and password file <none>
I20260809 00:56:08.147930 371 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:08.148103 371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:08.148476 371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:56:08.152326 371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "863d6bfc878544068e82693fdb0f1c0f"
format_stamp: "Formatted at 2026-08-09 00:56:08 on dist-test-slave-j2vl"
server_key: "4c07c22e534eaf6c5d28e5e8ef1d01da"
server_key_iv: "aa0986f4e363b3c884e5adcab7623a7f"
server_key_version: "encryptionkey@0"
I20260809 00:56:08.153582 371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "863d6bfc878544068e82693fdb0f1c0f"
format_stamp: "Formatted at 2026-08-09 00:56:08 on dist-test-slave-j2vl"
server_key: "4c07c22e534eaf6c5d28e5e8ef1d01da"
server_key_iv: "aa0986f4e363b3c884e5adcab7623a7f"
server_key_version: "encryptionkey@0"
I20260809 00:56:08.160554 371 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260809 00:56:08.164971 389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:08.166850 371 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:56:08.167037 371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "863d6bfc878544068e82693fdb0f1c0f"
format_stamp: "Formatted at 2026-08-09 00:56:08 on dist-test-slave-j2vl"
server_key: "4c07c22e534eaf6c5d28e5e8ef1d01da"
server_key_iv: "aa0986f4e363b3c884e5adcab7623a7f"
server_key_version: "encryptionkey@0"
I20260809 00:56:08.167255 371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:08.190253 371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:08.195001 371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:08.195284 371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:08.196309 371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:08.197914 371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:08.197996 371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:08.198089 371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:08.198134 371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:08.235194 371 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:35855
I20260809 00:56:08.235313 558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:35855 every 8 connection(s)
I20260809 00:56:08.236661 371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:56:08.243111 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 371
I20260809 00:56:08.243343 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:56:08.243976 10116 external_mini_cluster.cc:1428] Setting key 662de804796485467702cfc2c5372bf0
Aug 09 00:56:08 dist-test-slave-j2vl krb5kdc[32452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236968, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:08.264865 561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:08.239596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:42861 (local address 127.9.225.62:35871)
0809 00:56:08.240110 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:08.240117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:08.241280 (+ 1163us) server_negotiation.cc:407] Connection header received
0809 00:56:08.243703 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:08.243709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:08.243797 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:08.243909 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:08.246241 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.247071 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.250031 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.250528 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.250657 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:08.256553 (+ 5896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:08.256564 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:08.256573 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:08.256634 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:08.261052 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:08.262114 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:08.262119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:08.262121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:08.262264 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:08.262795 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:08.262801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:08.262803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:08.263126 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:08.263300 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:08.264169 (+ 869us) server_negotiation.cc:299] Negotiation successful
0809 00:56:08.264472 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":214,"threads_started":1}
I20260809 00:56:08.266039 562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:08.240270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:42861)
0809 00:56:08.241092 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:08.241137 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:08.243254 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:08.244283 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:08.244296 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:08.245046 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:08.245962 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:08.245991 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.247275 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.247283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:08.249776 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:08.249798 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.250831 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.250841 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:08.251260 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:08.252285 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:08.252334 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:08.256131 (+ 3797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:08.261291 (+ 5160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:08.261302 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:08.261322 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:08.261935 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:08.262407 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:08.262413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:08.262416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:08.262643 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:08.263269 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:08.263279 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:08.263723 (+ 444us) client_negotiation.cc:769] Sending connection context
0809 00:56:08.264039 (+ 316us) client_negotiation.cc:240] Negotiation successful
0809 00:56:08.264686 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":302,"threads_started":1}
I20260809 00:56:08.267907 559 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35871
I20260809 00:56:08.268281 559 heartbeater.cc:464] Registering TS with master...
I20260809 00:56:08.269157 559 heartbeater.cc:511] Master 127.9.225.62:35871 requested a full tablet report, sending...
I20260809 00:56:08.271307 32528 ts_manager.cc:194] Registered new tserver with Master: 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855)
I20260809 00:56:08.272352 32528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:42861
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:56:08.321854 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35871
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:56:08.674031 357 heartbeater.cc:503] Master 127.9.225.62:35871 was elected leader, sending a full tablet report...
W20260809 00:56:08.707042 566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:08.707458 566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:08.707636 566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:08.715328 566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:08.715530 566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:08.715724 566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:56:08.726445 566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:08.728228 566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:08.729853 566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:08.744856 581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:08.744921 584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:08.744956 577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:08.746227 566 server_base.cc:1034] running on GCE node
I20260809 00:56:08.746999 566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:08.748171 566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:08.749462 566 hybrid_clock.cc:648] HybridClock initialized: now 1786236968749407 us; error 52 us; skew 500 ppm
I20260809 00:56:08.749838 566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:08 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236968, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:08.753587 566 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:56:08.755363 566 webserver.cc:460] Webserver started at http://127.9.225.3:41689/ using document root <none> and password file <none>
I20260809 00:56:08.756150 566 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:08.756270 566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:08.756567 566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:56:08.759980 566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "a2f297c7bfdf4e5db0e41af3640a4202"
format_stamp: "Formatted at 2026-08-09 00:56:08 on dist-test-slave-j2vl"
server_key: "b1a2c929fcdf62a8981ffb444189b006"
server_key_iv: "a512616f7af1e55c723841fb3218a3d7"
server_key_version: "encryptionkey@0"
I20260809 00:56:08.760845 566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "a2f297c7bfdf4e5db0e41af3640a4202"
format_stamp: "Formatted at 2026-08-09 00:56:08 on dist-test-slave-j2vl"
server_key: "b1a2c929fcdf62a8981ffb444189b006"
server_key_iv: "a512616f7af1e55c723841fb3218a3d7"
server_key_version: "encryptionkey@0"
I20260809 00:56:08.767172 566 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260809 00:56:08.771905 595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:08.774013 566 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 00:56:08.774209 566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "a2f297c7bfdf4e5db0e41af3640a4202"
format_stamp: "Formatted at 2026-08-09 00:56:08 on dist-test-slave-j2vl"
server_key: "b1a2c929fcdf62a8981ffb444189b006"
server_key_iv: "a512616f7af1e55c723841fb3218a3d7"
server_key_version: "encryptionkey@0"
I20260809 00:56:08.774436 566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:08.796613 566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:08.801901 566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:08.802301 566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:08.803391 566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:08.805130 566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:08.805215 566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:08.805313 566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:08.805361 566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:08.842276 566 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:37947
I20260809 00:56:08.842336 782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:37947 every 8 connection(s)
I20260809 00:56:08.843709 566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:56:08.846189 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 566
I20260809 00:56:08.846415 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:56:08.847065 10116 external_mini_cluster.cc:1428] Setting key 9b88e303d6f54882b235d16e6ba39a2c
Aug 09 00:56:08 dist-test-slave-j2vl krb5kdc[32452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236968, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:08.868469 794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:08.846907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:35323 (local address 127.9.225.62:35871)
0809 00:56:08.847476 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:08.847482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:08.848743 (+ 1261us) server_negotiation.cc:407] Connection header received
0809 00:56:08.850754 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:08.850759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:08.850828 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:08.850925 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:08.852584 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.853515 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.855344 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.856202 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.856330 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:08.860256 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:08.860261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:08.860264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:08.860305 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:08.864708 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:08.865717 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:08.865726 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:08.865730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:08.865887 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:08.866398 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:08.866404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:08.866408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:08.866727 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:08.866931 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:08.867753 (+ 822us) server_negotiation.cc:299] Negotiation successful
0809 00:56:08.868042 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":293,"threads_started":1}
I20260809 00:56:08.869861 796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:08.847576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:35323)
0809 00:56:08.848553 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:08.848613 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:08.850485 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:08.851162 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:08.851170 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:08.851736 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:08.852386 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:08.852406 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.853697 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.853702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:08.855117 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:08.855132 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.856474 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.856479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:08.856702 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:08.857549 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:08.857580 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:08.859972 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:08.864901 (+ 4929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:08.864920 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:08.864939 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:08.865576 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:08.866024 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:08.866029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:08.866034 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:08.866262 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:08.866883 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:08.866894 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:08.867315 (+ 421us) client_negotiation.cc:769] Sending connection context
0809 00:56:08.867634 (+ 319us) client_negotiation.cc:240] Negotiation successful
0809 00:56:08.868316 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":392,"threads_started":1}
I20260809 00:56:08.871878 790 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35871
I20260809 00:56:08.872334 790 heartbeater.cc:464] Registering TS with master...
I20260809 00:56:08.873415 790 heartbeater.cc:511] Master 127.9.225.62:35871 requested a full tablet report, sending...
I20260809 00:56:08.875432 32528 ts_manager.cc:194] Registered new tserver with Master: a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947)
I20260809 00:56:08.876804 32528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:35323
I20260809 00:56:08.879829 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:56:08.912495 794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:08.895735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.9.225.62:35871)
0809 00:56:08.895955 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:08.895960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:08.896369 (+ 409us) server_negotiation.cc:407] Connection header received
0809 00:56:08.896589 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:08.896593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:08.896664 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:08.896808 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:08.898602 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.899392 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.900957 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.901509 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.901665 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:08.904570 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:08.904580 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:08.904587 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:08.904646 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:08.908797 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:08.909855 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:08.909863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:08.909869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:08.910075 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:08.910489 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:08.910497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:08.910501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:08.911085 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:08.911397 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:08.911730 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 00:56:08.912047 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 00:56:08.924669 32528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:56:08.934005 32528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:56:08.983328 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:08.951049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.9.225.3:37947)
0809 00:56:08.961146 (+ 10097us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:08.961162 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:08.965288 (+ 4126us) server_negotiation.cc:407] Connection header received
0809 00:56:08.965466 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:08.965480 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:08.965839 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:08.966091 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:08.969257 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.973034 (+ 3777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.978234 (+ 5200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.981512 (+ 3278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.981619 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:08.981909 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:08.982723 (+ 814us) server_negotiation.cc:299] Negotiation successful
0809 00:56:08.982925 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9776,"thread_start_us":14332,"threads_started":1}
I20260809 00:56:08.983361 818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:08.950655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:37947 (local address 127.0.0.1:60902)
0809 00:56:08.964589 (+ 13934us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:08.964671 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:08.965000 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:08.968130 (+ 3130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:08.968143 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:08.968231 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:08.968848 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:08.968867 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.973260 (+ 4393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.973269 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:08.977940 (+ 4671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:08.977971 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.982193 (+ 4222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.982201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:08.982428 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:08.982478 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 00:56:08.982637 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 00:56:08.982883 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13478,"thread_start_us":15256,"threads_started":1}
I20260809 00:56:08.988600 676 tablet_service.cc:1467] Processing CreateTablet for tablet 888af009fef04364b71dda82f46e69d0 (DEFAULT_TABLE table=test-table [id=00cd83b81149420a8f177f8573fad458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:56:08.990696 676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888af009fef04364b71dda82f46e69d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:08.990983 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:08.951000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:35855 (local address 127.0.0.1:38108)
0809 00:56:08.966238 (+ 15238us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:08.966263 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:08.966415 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:08.980625 (+ 14210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:08.980629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:08.980656 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:08.981109 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:08.981125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.985446 (+ 4321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.985453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:08.987590 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:08.987611 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.989917 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.989924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:08.990123 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:08.990173 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 00:56:08.990299 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 00:56:08.990502 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15099,"mutex_wait_us":1,"thread_start_us":26719,"threads_started":1}
I20260809 00:56:08.991933 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:08.951437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.9.225.2:35855)
0809 00:56:08.979932 (+ 28495us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:08.979943 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:08.979978 (+ 35us) server_negotiation.cc:407] Connection header received
0809 00:56:08.980088 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:08.980095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:08.980376 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:08.980599 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:08.983638 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.985200 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.987818 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.989329 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.989496 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:08.989732 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:08.991404 (+ 1672us) server_negotiation.cc:299] Negotiation successful
0809 00:56:08.991606 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28278,"thread_start_us":29330,"threads_started":1}
I20260809 00:56:08.992285 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:08.952290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.9.225.1:38923)
0809 00:56:08.977146 (+ 24856us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:08.977161 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:08.978413 (+ 1252us) server_negotiation.cc:407] Connection header received
0809 00:56:08.978576 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:08.978589 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:08.978909 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:08.979168 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:08.980099 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.981786 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.985860 (+ 4074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.989632 (+ 3772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.989801 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:08.990073 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:08.991664 (+ 1591us) server_negotiation.cc:299] Negotiation successful
0809 00:56:08.991888 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24547,"thread_start_us":24880,"threads_started":1}
I20260809 00:56:08.991494 816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:08.951324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38923 (local address 127.0.0.1:37662)
0809 00:56:08.978033 (+ 26709us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:08.978053 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:08.978232 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:08.979133 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:08.979138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:08.979172 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:08.979779 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:08.979796 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.981935 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.981944 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:08.985613 (+ 3669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:08.985638 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:08.990264 (+ 4626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:08.990272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:08.990561 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:08.990607 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 00:56:08.990709 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 00:56:08.991030 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26566,"mutex_wait_us":31,"thread_start_us":12575,"threads_started":1}
I20260809 00:56:08.996937 463 tablet_service.cc:1467] Processing CreateTablet for tablet 888af009fef04364b71dda82f46e69d0 (DEFAULT_TABLE table=test-table [id=00cd83b81149420a8f177f8573fad458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:56:08.998785 463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888af009fef04364b71dda82f46e69d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:09.007826 821 tablet_bootstrap.cc:492] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202: Bootstrap starting.
I20260809 00:56:09.008405 32715 tablet_service.cc:1467] Processing CreateTablet for tablet 888af009fef04364b71dda82f46e69d0 (DEFAULT_TABLE table=test-table [id=00cd83b81149420a8f177f8573fad458]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:56:09.010052 32715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888af009fef04364b71dda82f46e69d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:09.013607 821 tablet_bootstrap.cc:654] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:09.015322 821 log.cc:824] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:09.025861 821 tablet_bootstrap.cc:492] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202: No bootstrap required, opened a new log
I20260809 00:56:09.026288 821 ts_tablet_manager.cc:1397] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20260809 00:56:09.030339 824 tablet_bootstrap.cc:492] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f: Bootstrap starting.
I20260809 00:56:09.031646 825 tablet_bootstrap.cc:492] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe: Bootstrap starting.
I20260809 00:56:09.034663 824 tablet_bootstrap.cc:654] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:09.035583 825 tablet_bootstrap.cc:654] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:09.036120 824 log.cc:824] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:09.036789 825 log.cc:824] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:09.038597 821 raft_consensus.cc:348] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } }
I20260809 00:56:09.039038 821 raft_consensus.cc:374] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:09.039137 821 raft_consensus.cc:729] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f297c7bfdf4e5db0e41af3640a4202, State: Initialized, Role: FOLLOWER
I20260809 00:56:09.039827 821 consensus_queue.cc:260] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } }
I20260809 00:56:09.040005 824 tablet_bootstrap.cc:492] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f: No bootstrap required, opened a new log
I20260809 00:56:09.040076 825 tablet_bootstrap.cc:492] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe: No bootstrap required, opened a new log
I20260809 00:56:09.040354 824 ts_tablet_manager.cc:1397] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 00:56:09.040352 825 ts_tablet_manager.cc:1397] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 00:56:09.041126 790 heartbeater.cc:503] Master 127.9.225.62:35871 was elected leader, sending a full tablet report...
I20260809 00:56:09.041878 821 ts_tablet_manager.cc:1428] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260809 00:56:09.052947 825 raft_consensus.cc:348] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } }
I20260809 00:56:09.052942 824 raft_consensus.cc:348] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } }
I20260809 00:56:09.053593 825 raft_consensus.cc:374] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:09.053593 824 raft_consensus.cc:374] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:09.053714 824 raft_consensus.cc:729] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863d6bfc878544068e82693fdb0f1c0f, State: Initialized, Role: FOLLOWER
I20260809 00:56:09.053730 825 raft_consensus.cc:729] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156cb7dab3294f11a63a1cb8e16f5afe, State: Initialized, Role: FOLLOWER
I20260809 00:56:09.054514 825 consensus_queue.cc:260] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } }
I20260809 00:56:09.054517 824 consensus_queue.cc:260] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } }
I20260809 00:56:09.055974 559 heartbeater.cc:503] Master 127.9.225.62:35871 was elected leader, sending a full tablet report...
I20260809 00:56:09.057199 825 ts_tablet_manager.cc:1428] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260809 00:56:09.057205 824 ts_tablet_manager.cc:1428] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
W20260809 00:56:09.095594 792 tablet.cc:2369] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:56:09.113682 831 raft_consensus.cc:482] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:56:09.114018 831 raft_consensus.cc:504] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } }
I20260809 00:56:09.116003 831 leader_election.cc:288] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855), 156cb7dab3294f11a63a1cb8e16f5afe (127.9.225.1:38923)
I20260809 00:56:09.122574 796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:09.116312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38923 (local address 127.9.225.3:51211)
0809 00:56:09.116634 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:09.116657 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:09.116807 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:09.117272 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:09.117277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:09.117304 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:09.117531 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:09.117540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.118614 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.118619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:09.120184 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:09.120199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.121955 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.121959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:09.122104 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:09.122130 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 00:56:09.122213 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 00:56:09.122335 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":59}
I20260809 00:56:09.122711 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:09.116736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:51211 (local address 127.9.225.1:38923)
0809 00:56:09.116980 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:09.116984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:09.117001 (+ 17us) server_negotiation.cc:407] Connection header received
0809 00:56:09.117067 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:09.117071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:09.117128 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:09.117242 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:09.117699 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.118456 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.120362 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.121776 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.121927 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:09.122048 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:09.122276 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 00:56:09.122439 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 00:56:09.123883 834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:09.116493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:35855 (local address 127.9.225.3:35669)
0809 00:56:09.117479 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:09.117495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:09.117648 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:09.117991 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:09.117996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:09.118019 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:09.118342 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:09.118351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.119395 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.119400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:09.121088 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:09.121100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.123110 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.123117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:09.123303 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:09.123339 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 00:56:09.123450 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 00:56:09.123557 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":51,"thread_start_us":555,"threads_started":1}
I20260809 00:56:09.123943 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:09.116560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:35669 (local address 127.9.225.2:35855)
0809 00:56:09.116841 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:09.116849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:09.117596 (+ 747us) server_negotiation.cc:407] Connection header received
0809 00:56:09.117809 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:09.117814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:09.117873 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:09.117976 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:09.118519 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.119225 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.121286 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.122525 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.122670 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:09.122778 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:09.123498 (+ 720us) server_negotiation.cc:299] Negotiation successful
0809 00:56:09.123658 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 00:56:09.124029 32753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888af009fef04364b71dda82f46e69d0" candidate_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" is_pre_election: true
I20260809 00:56:09.124745 32753 raft_consensus.cc:2454] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f297c7bfdf4e5db0e41af3640a4202 in term 0.
I20260809 00:56:09.125041 490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888af009fef04364b71dda82f46e69d0" candidate_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863d6bfc878544068e82693fdb0f1c0f" is_pre_election: true
I20260809 00:56:09.125653 490 raft_consensus.cc:2454] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f297c7bfdf4e5db0e41af3640a4202 in term 0.
I20260809 00:56:09.125603 600 leader_election.cc:302] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156cb7dab3294f11a63a1cb8e16f5afe, a2f297c7bfdf4e5db0e41af3640a4202; no voters:
I20260809 00:56:09.126199 831 raft_consensus.cc:2781] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:56:09.126406 831 raft_consensus.cc:482] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:56:09.126546 831 raft_consensus.cc:3037] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:09.129070 831 raft_consensus.cc:504] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } }
I20260809 00:56:09.130038 831 leader_election.cc:288] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [CANDIDATE]: Term 1 election: Requested vote from peers 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855), 156cb7dab3294f11a63a1cb8e16f5afe (127.9.225.1:38923)
I20260809 00:56:09.130690 490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888af009fef04364b71dda82f46e69d0" candidate_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863d6bfc878544068e82693fdb0f1c0f"
I20260809 00:56:09.130787 32753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888af009fef04364b71dda82f46e69d0" candidate_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "156cb7dab3294f11a63a1cb8e16f5afe"
I20260809 00:56:09.130954 490 raft_consensus.cc:3037] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:09.131105 32753 raft_consensus.cc:3037] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:09.133662 32753 raft_consensus.cc:2454] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f297c7bfdf4e5db0e41af3640a4202 in term 1.
I20260809 00:56:09.133946 490 raft_consensus.cc:2454] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f297c7bfdf4e5db0e41af3640a4202 in term 1.
I20260809 00:56:09.134298 600 leader_election.cc:302] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156cb7dab3294f11a63a1cb8e16f5afe, a2f297c7bfdf4e5db0e41af3640a4202; no voters:
I20260809 00:56:09.134759 831 raft_consensus.cc:2781] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:56:09.135231 831 raft_consensus.cc:686] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 1 LEADER]: Becoming Leader. State: Replica: a2f297c7bfdf4e5db0e41af3640a4202, State: Running, Role: LEADER
I20260809 00:56:09.135787 831 consensus_queue.cc:237] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } }
I20260809 00:56:09.142078 32526 catalog_manager.cc:5495] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3). New cstate: current_term: 1 leader_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } health_report { overall_health: UNKNOWN } } }
W20260809 00:56:09.143225 362 tablet.cc:2369] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:56:09.173938 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:09.167903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.9.225.3:37947)
0809 00:56:09.168104 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:09.168109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:09.168130 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:56:09.168317 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:09.168321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:09.168429 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:09.168569 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:56:09.169032 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.169864 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.171379 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.171936 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.172057 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:09.172655 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:56:09.173307 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:56:09.173467 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:09.173621 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 00:56:09.173708 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 00:56:09.183687 490 raft_consensus.cc:1261] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f [term 1 FOLLOWER]: Refusing update from remote peer a2f297c7bfdf4e5db0e41af3640a4202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:56:09.183687 32753 raft_consensus.cc:1261] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Refusing update from remote peer a2f297c7bfdf4e5db0e41af3640a4202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:56:09.184837 835 consensus_queue.cc:1035] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:56:09.185254 831 consensus_queue.cc:1035] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:56:09.198805 840 mvcc.cc:205] Tried to move back new op lower bound from 7316426625764016128 to 7316426625584574464. Current Snapshot: MvccSnapshot[applied={T|T < 7316426625764016128}]
I20260809 00:56:09.203460 843 mvcc.cc:205] Tried to move back new op lower bound from 7316426625764016128 to 7316426625584574464. Current Snapshot: MvccSnapshot[applied={T|T < 7316426625764016128}]
I20260809 00:56:09.220743 32526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:56:09.221123 32526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:56:09.226933 32526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Sending DeleteTablet for 3 replicas of tablet 888af009fef04364b71dda82f46e69d0
I20260809 00:56:09.228566 32715 tablet_service.cc:1514] Processing DeleteTablet for tablet 888af009fef04364b71dda82f46e69d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:56:09 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:37662
I20260809 00:56:09.228576 676 tablet_service.cc:1514] Processing DeleteTablet for tablet 888af009fef04364b71dda82f46e69d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:56:09 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:60902
I20260809 00:56:09.229434 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 32622
I20260809 00:56:09.229498 463 tablet_service.cc:1514] Processing DeleteTablet for tablet 888af009fef04364b71dda82f46e69d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:56:09 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:38108
I20260809 00:56:09.230630 862 tablet_replica.cc:330] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe: stopping tablet replica
I20260809 00:56:09.230924 861 tablet_replica.cc:330] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202: stopping tablet replica
I20260809 00:56:09.231230 862 raft_consensus.cc:2229] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:56:09.231499 861 raft_consensus.cc:2229] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:56:09.231922 862 raft_consensus.cc:2258] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:56:09.232445 861 raft_consensus.cc:2258] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:56:09.234738 862 ts_tablet_manager.cc:1905] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:56:09.239034 863 tablet_replica.cc:330] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f: stopping tablet replica
I20260809 00:56:09.241734 863 raft_consensus.cc:2229] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:56:09.242341 863 raft_consensus.cc:2258] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:56:09.242674 861 ts_tablet_manager.cc:1905] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:56:09.243136 862 ts_tablet_manager.cc:1918] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:56:09.243409 862 log.cc:1196] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/888af009fef04364b71dda82f46e69d0
I20260809 00:56:09.244082 862 ts_tablet_manager.cc:1939] T 888af009fef04364b71dda82f46e69d0 P 156cb7dab3294f11a63a1cb8e16f5afe: Deleting consensus metadata
I20260809 00:56:09.246325 32501 catalog_manager.cc:4841] TS 156cb7dab3294f11a63a1cb8e16f5afe (127.9.225.1:38923): tablet 888af009fef04364b71dda82f46e69d0 (table test-table [id=00cd83b81149420a8f177f8573fad458]) successfully deleted
I20260809 00:56:09.248346 863 ts_tablet_manager.cc:1905] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 00:56:09 dist-test-slave-j2vl krb5kdc[32452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:56:09.256665 863 ts_tablet_manager.cc:1918] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:56:09.256891 863 log.cc:1196] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/888af009fef04364b71dda82f46e69d0
I20260809 00:56:09.257740 863 ts_tablet_manager.cc:1939] T 888af009fef04364b71dda82f46e69d0 P 863d6bfc878544068e82693fdb0f1c0f: Deleting consensus metadata
I20260809 00:56:09.259464 861 ts_tablet_manager.cc:1918] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:56:09.260346 32502 catalog_manager.cc:4841] TS 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855): tablet 888af009fef04364b71dda82f46e69d0 (table test-table [id=00cd83b81149420a8f177f8573fad458]) successfully deleted
I20260809 00:56:09.261031 861 log.cc:1196] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/888af009fef04364b71dda82f46e69d0
I20260809 00:56:09.261790 861 ts_tablet_manager.cc:1939] T 888af009fef04364b71dda82f46e69d0 P a2f297c7bfdf4e5db0e41af3640a4202: Deleting consensus metadata
I20260809 00:56:09.262478 820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:09.230999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.9.225.1:38923)
0809 00:56:09.231268 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:09.231278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:09.238176 (+ 6898us) server_negotiation.cc:407] Connection header received
0809 00:56:09.238399 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:09.238405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:09.238495 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:09.238610 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:09.243186 (+ 4576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.244443 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.247037 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.247720 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.247820 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:09.253897 (+ 6077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:09.253925 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:09.253951 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:09.254009 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:09.258262 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:09.259451 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:09.259464 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:09.259468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:09.259670 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:09.260140 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:09.260145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:09.260149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:09.260678 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:09.261044 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:09.261797 (+ 753us) server_negotiation.cc:299] Negotiation successful
0809 00:56:09.262068 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 00:56:09.263864 32498 catalog_manager.cc:4841] TS a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947): tablet 888af009fef04364b71dda82f46e69d0 (table test-table [id=00cd83b81149420a8f177f8573fad458]) successfully deleted
I20260809 00:56:09.264051 351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:09.375103 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 32622
I20260809 00:56:09.403002 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 371
Aug 09 00:56:09 dist-test-slave-j2vl krb5kdc[32452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:56:09.422257 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:09.404036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.9.225.2:35855)
0809 00:56:09.404230 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:09.404236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:09.404257 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:56:09.404330 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:09.404335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:09.404427 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:09.404547 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:09.407133 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.407946 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.409396 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.409826 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.409974 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:09.415256 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:09.415278 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:09.415297 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:09.415346 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:09.418862 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:09.419773 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:09.419783 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:09.419786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:09.419942 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:09.420388 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:09.420394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:09.420404 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:09.420846 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:09.421201 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:09.421620 (+ 419us) server_negotiation.cc:299] Negotiation successful
0809 00:56:09.421889 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 00:56:09.423197 552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:09.518491 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 371
I20260809 00:56:09.561101 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 566
Aug 09 00:56:09 dist-test-slave-j2vl krb5kdc[32452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:56:09.581979 817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:09.562057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.9.225.3:37947)
0809 00:56:09.562268 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:09.562275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:09.562298 (+ 23us) server_negotiation.cc:407] Connection header received
0809 00:56:09.562438 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:09.562443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:09.562529 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:09.562663 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:09.564068 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.565314 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.566847 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:09.567520 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:09.567693 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:09.572974 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:09.573001 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:09.573027 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:09.573107 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:09.577946 (+ 4839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:09.579071 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:09.579083 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:09.579086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:09.579333 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:09.579781 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:09.579788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:09.579791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:09.580365 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:09.580791 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:09.581174 (+ 383us) server_negotiation.cc:299] Negotiation successful
0809 00:56:09.581462 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 00:56:09.582939 775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:09.714680 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 566
I20260809 00:56:09.751245 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 32478
I20260809 00:56:09.752615 32573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:09.872118 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 32478
I20260809 00:56:09.908321 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35871
--webserver_interface=127.9.225.62
--webserver_port=43677
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:35871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:56:10.291636 871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:10.292088 871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:10.292265 871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:10.300053 871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:56:10.300274 871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:10.300375 871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:10.300489 871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:56:10.300565 871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:56:10.313133 871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:35871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=43677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:10.314832 871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:10.316422 871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:10.347564 881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:10.347515 878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:10.347445 879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:10.348462 871 server_base.cc:1034] running on GCE node
I20260809 00:56:10.349606 871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:10.351684 871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:10.352952 871 hybrid_clock.cc:648] HybridClock initialized: now 1786236970352889 us; error 55 us; skew 500 ppm
I20260809 00:56:10.353544 871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:10 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236970, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:10.358050 871 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:56:10.360077 871 webserver.cc:460] Webserver started at http://127.9.225.62:43677/ using document root <none> and password file <none>
I20260809 00:56:10.361001 871 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:10.361150 871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:10.368830 871 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 00:56:10.373118 894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:10.375679 871 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260809 00:56:10.375972 871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "25d7d397d6734f85a2f483ee67352746"
format_stamp: "Formatted at 2026-08-09 00:56:06 on dist-test-slave-j2vl"
server_key: "2ddd14e9279df414455ac81771e4af87"
server_key_iv: "0f81b8ba5af581ba62706611f8a53739"
server_key_version: "encryptionkey@0"
I20260809 00:56:10.377314 871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:10.399250 871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:10.405901 871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:10.406287 871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:10.429776 871 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:35871
I20260809 00:56:10.429801 984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:35871 every 8 connection(s)
I20260809 00:56:10.431624 871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:56:10.434206 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 871
I20260809 00:56:10.435143 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:38923
--local_ip_for_outbound_sockets=127.9.225.1
--tserver_master_addrs=127.9.225.62:35871
--webserver_port=44941
--webserver_interface=127.9.225.1
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:56:10.440246 985 sys_catalog.cc:263] Verifying existing consensus state
I20260809 00:56:10.444193 985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Bootstrap starting.
I20260809 00:56:10.476032 985 log.cc:824] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:10.494908 985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 00:56:10.495710 985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Bootstrap complete.
I20260809 00:56:10.510867 985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } }
I20260809 00:56:10.512239 985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d7d397d6734f85a2f483ee67352746, State: Initialized, Role: FOLLOWER
I20260809 00:56:10.513082 985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } }
I20260809 00:56:10.513460 985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:56:10.513608 985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:56:10.513820 985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 1 FOLLOWER]: Advancing to term 2
I20260809 00:56:10.521301 985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } }
I20260809 00:56:10.522236 985 leader_election.cc:302] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25d7d397d6734f85a2f483ee67352746; no voters:
I20260809 00:56:10.523003 985 leader_election.cc:288] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 00:56:10.523217 992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 00:56:10.524755 992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 2 LEADER]: Becoming Leader. State: Replica: 25d7d397d6734f85a2f483ee67352746, State: Running, Role: LEADER
I20260809 00:56:10.525506 992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } }
I20260809 00:56:10.525904 985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:56:10.528774 995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25d7d397d6734f85a2f483ee67352746. Latest consensus state: current_term: 2 leader_uuid: "25d7d397d6734f85a2f483ee67352746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } } }
I20260809 00:56:10.528688 994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "25d7d397d6734f85a2f483ee67352746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } } }
I20260809 00:56:10.529160 994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:10.529165 995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:10.543694 999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:56:10.553267 999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:56:10.555048 999 catalog_manager.cc:1259] Loaded cluster ID: ee8a326af34041dda221f75293b6c567
I20260809 00:56:10.555251 999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:56:10.559657 999 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 00:56:10.578298 999 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 00:56:10.588032 999 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:56:10.589676 999 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Loaded TSK: 0
I20260809 00:56:10.590988 999 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 00:56:10.901618 990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:10.902036 990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:10.902215 990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:10.910739 990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:10.910938 990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:10.911130 990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:56:10.922677 990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:38923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=44941
--tserver_master_addrs=127.9.225.62:35871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:10.924290 990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:10.926227 990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:10.943229 1026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:10.943274 1027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:10.944108 1030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:10.944216 990 server_base.cc:1034] running on GCE node
I20260809 00:56:10.945320 990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:10.946762 990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:10.948050 990 hybrid_clock.cc:648] HybridClock initialized: now 1786236970947963 us; error 73 us; skew 500 ppm
I20260809 00:56:10.948469 990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:10 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236970, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:10.953248 990 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:56:10.955482 990 webserver.cc:460] Webserver started at http://127.9.225.1:44941/ using document root <none> and password file <none>
I20260809 00:56:10.956563 990 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:10.956776 990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:10.965754 990 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 00:56:10.969926 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:10.971881 990 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 00:56:10.972110 990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "156cb7dab3294f11a63a1cb8e16f5afe"
format_stamp: "Formatted at 2026-08-09 00:56:07 on dist-test-slave-j2vl"
server_key: "0dcbe8098e776157e256bf5babda35e6"
server_key_iv: "d78ac24cb8d98107e95f64bad47f43b4"
server_key_version: "encryptionkey@0"
I20260809 00:56:10.973658 990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:10.994777 990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:10.999877 990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:11.000166 990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:11.001303 990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:11.002903 990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:11.003003 990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:11.003093 990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:11.003152 990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:11.042348 990 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:38923
I20260809 00:56:11.042428 1242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:38923 every 8 connection(s)
I20260809 00:56:11.044057 990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:56:11.048801 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 990
I20260809 00:56:11.049633 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:35855
--local_ip_for_outbound_sockets=127.9.225.2
--tserver_master_addrs=127.9.225.62:35871
--webserver_port=41007
--webserver_interface=127.9.225.2
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:56:11 dist-test-slave-j2vl krb5kdc[32452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236970, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:11.073076 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:11.047618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:35413 (local address 127.9.225.62:35871)
0809 00:56:11.048377 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:11.048408 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:11.049077 (+ 669us) server_negotiation.cc:407] Connection header received
0809 00:56:11.051609 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:11.051622 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:11.052021 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:11.052443 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:11.054239 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:11.055543 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:11.057250 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:11.058039 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:11.058273 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:11.063461 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:11.063472 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:11.063491 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:11.063547 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:11.067429 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:11.068459 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:11.068468 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:11.068474 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:11.068673 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:11.069204 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:11.069209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:11.069211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:11.069937 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:11.070200 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:11.071232 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0809 00:56:11.071682 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":327,"threads_started":1}
I20260809 00:56:11.073114 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:11.047809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:35413)
0809 00:56:11.048815 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:11.048867 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:11.050951 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:11.052625 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:11.052640 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:11.053400 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:11.054000 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:11.054021 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:11.055758 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:11.055763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:11.057068 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:11.057084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:11.058318 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:11.058327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:11.058775 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:11.059679 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:11.059706 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:11.063047 (+ 3341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:11.067684 (+ 4637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:11.067697 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:11.067724 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:11.068262 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:11.068819 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:11.068824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:11.068828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:11.069055 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:11.070292 (+ 1237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:11.070302 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:11.070676 (+ 374us) client_negotiation.cc:769] Sending connection context
0809 00:56:11.070974 (+ 298us) client_negotiation.cc:240] Negotiation successful
0809 00:56:11.071654 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":454,"threads_started":1}
I20260809 00:56:11.076115 1243 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35871
I20260809 00:56:11.076469 1243 heartbeater.cc:464] Registering TS with master...
I20260809 00:56:11.077315 1243 heartbeater.cc:511] Master 127.9.225.62:35871 requested a full tablet report, sending...
I20260809 00:56:11.080065 923 ts_manager.cc:194] Registered new tserver with Master: 156cb7dab3294f11a63a1cb8e16f5afe (127.9.225.1:38923)
I20260809 00:56:11.081772 923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:35413
W20260809 00:56:11.431288 1247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:11.431655 1247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:11.431840 1247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:11.439649 1247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:11.439846 1247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:11.440059 1247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:56:11.450896 1247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:35855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=41007
--tserver_master_addrs=127.9.225.62:35871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:11.452581 1247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:11.454135 1247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:11.469692 1258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:11.469715 1263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:11.469638 1259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:11.471040 1247 server_base.cc:1034] running on GCE node
I20260809 00:56:11.471850 1247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:11.473021 1247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:11.474280 1247 hybrid_clock.cc:648] HybridClock initialized: now 1786236971474231 us; error 73 us; skew 500 ppm
I20260809 00:56:11.474653 1247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:11 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236971, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:11.478790 1247 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:56:11.480692 1247 webserver.cc:460] Webserver started at http://127.9.225.2:41007/ using document root <none> and password file <none>
I20260809 00:56:11.481547 1247 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:11.481686 1247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:11.488291 1247 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 00:56:11.491825 1274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:11.493773 1247 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:56:11.494161 1247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "863d6bfc878544068e82693fdb0f1c0f"
format_stamp: "Formatted at 2026-08-09 00:56:08 on dist-test-slave-j2vl"
server_key: "4c07c22e534eaf6c5d28e5e8ef1d01da"
server_key_iv: "aa0986f4e363b3c884e5adcab7623a7f"
server_key_version: "encryptionkey@0"
I20260809 00:56:11.495471 1247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:11.533041 1247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:11.538153 1247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:11.538493 1247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:11.539599 1247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:11.541049 1247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:11.541145 1247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:11.541236 1247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:11.541294 1247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:11.577555 1247 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:35855
I20260809 00:56:11.577661 1446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:35855 every 8 connection(s)
I20260809 00:56:11.579255 1247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:56:11.585821 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 1247
I20260809 00:56:11.586629 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:37947
--local_ip_for_outbound_sockets=127.9.225.3
--tserver_master_addrs=127.9.225.62:35871
--webserver_port=41689
--webserver_interface=127.9.225.3
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:56:11 dist-test-slave-j2vl krb5kdc[32452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236971, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:11.607491 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:11.582407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:55781 (local address 127.9.225.62:35871)
0809 00:56:11.582980 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:11.582985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:11.584303 (+ 1318us) server_negotiation.cc:407] Connection header received
0809 00:56:11.586975 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:11.586984 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:11.587092 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:11.587199 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:11.589749 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:11.590508 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:11.593329 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:11.593892 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:11.594020 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:11.599104 (+ 5084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:11.599111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:11.599114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:11.599157 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:11.603307 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:11.604320 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:11.604329 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:11.604335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:11.604514 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:11.605061 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:11.605069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:11.605075 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:11.605558 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:11.605763 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:11.606639 (+ 876us) server_negotiation.cc:299] Negotiation successful
0809 00:56:11.606975 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":248,"threads_started":1}
I20260809 00:56:11.608340 1454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:11.583128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:55781)
0809 00:56:11.584081 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:11.584125 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:11.586548 (+ 2423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:11.587546 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:11.587562 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:11.588492 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:11.589473 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:11.589509 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:11.590735 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:11.590743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:11.593085 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:11.593109 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:11.594200 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:11.594211 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:11.594615 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:11.595530 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:11.595570 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:11.598762 (+ 3192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:11.603531 (+ 4769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:11.603541 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:11.603558 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:11.604125 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:11.604668 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:11.604671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:11.604674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:11.604918 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:11.605743 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:11.605752 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:11.606178 (+ 426us) client_negotiation.cc:769] Sending connection context
0809 00:56:11.606495 (+ 317us) client_negotiation.cc:240] Negotiation successful
0809 00:56:11.606996 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":395,"threads_started":1}
I20260809 00:56:11.610318 1448 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35871
I20260809 00:56:11.610716 1448 heartbeater.cc:464] Registering TS with master...
I20260809 00:56:11.611698 1448 heartbeater.cc:511] Master 127.9.225.62:35871 requested a full tablet report, sending...
I20260809 00:56:11.613672 923 ts_manager.cc:194] Registered new tserver with Master: 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855)
I20260809 00:56:11.615077 923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:55781
W20260809 00:56:11.974001 1455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:11.974331 1455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:11.974435 1455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:11.982808 1455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:11.983007 1455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:11.983239 1455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:56:11.994387 1455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:37947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=41689
--tserver_master_addrs=127.9.225.62:35871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:11.995870 1455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:11.997319 1455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:12.012130 1466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:12.012073 1462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:12.012163 1463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:12.013163 1455 server_base.cc:1034] running on GCE node
I20260809 00:56:12.013933 1455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:12.015146 1455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:12.016381 1455 hybrid_clock.cc:648] HybridClock initialized: now 1786236972016325 us; error 47 us; skew 500 ppm
I20260809 00:56:12.016784 1455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:12 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236972, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:12.025043 1455 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:56:12.027158 1455 webserver.cc:460] Webserver started at http://127.9.225.3:41689/ using document root <none> and password file <none>
I20260809 00:56:12.027992 1455 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:12.028131 1455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:12.034420 1455 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260809 00:56:12.037906 1478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:12.039758 1455 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260809 00:56:12.039978 1455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "a2f297c7bfdf4e5db0e41af3640a4202"
format_stamp: "Formatted at 2026-08-09 00:56:08 on dist-test-slave-j2vl"
server_key: "b1a2c929fcdf62a8981ffb444189b006"
server_key_iv: "a512616f7af1e55c723841fb3218a3d7"
server_key_version: "encryptionkey@0"
I20260809 00:56:12.041231 1455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:12.066233 1455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:12.071404 1455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:12.071765 1455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:12.073038 1455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:12.074687 1455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:12.074862 1455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:12.075016 1455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:12.075109 1455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:12.085666 1243 heartbeater.cc:503] Master 127.9.225.62:35871 was elected leader, sending a full tablet report...
I20260809 00:56:12.114552 1455 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:37947
I20260809 00:56:12.114615 1662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:37947 every 8 connection(s)
I20260809 00:56:12.116281 1455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:56:12.116814 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 1455
Aug 09 00:56:12 dist-test-slave-j2vl krb5kdc[32452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236972, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:12.136035 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.118189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36982 (local address 127.9.225.62:35871)
0809 00:56:12.118778 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:12.118785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:12.118951 (+ 166us) server_negotiation.cc:407] Connection header received
0809 00:56:12.119217 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:12.119222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:12.119305 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:12.119440 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:12.121724 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.122580 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.124037 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.124500 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.124663 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.128491 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:12.128498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:12.128501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:12.128544 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:12.132034 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:12.133161 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:12.133170 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:12.133174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:12.133418 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:12.133909 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:12.133916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:12.133920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:12.134345 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:12.134564 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:12.135291 (+ 727us) server_negotiation.cc:299] Negotiation successful
0809 00:56:12.135564 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":313,"threads_started":1}
I20260809 00:56:12.141872 1672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.120280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:50431 (local address 127.9.225.62:35871)
0809 00:56:12.120924 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:12.120931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:12.122115 (+ 1184us) server_negotiation.cc:407] Connection header received
0809 00:56:12.124127 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:12.124132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:12.124249 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:12.124384 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:12.126274 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.127178 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.129103 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.129685 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.129832 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.134680 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:12.134688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:12.134693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:12.134747 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:12.138228 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:12.139223 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:12.139231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:12.139235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:12.139400 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:12.140014 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:12.140019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:12.140021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:12.140323 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:12.140448 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:12.141260 (+ 812us) server_negotiation.cc:299] Negotiation successful
0809 00:56:12.141560 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":310,"threads_started":1}
I20260809 00:56:12.142967 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.120976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:50431)
0809 00:56:12.121926 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.121988 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:12.123808 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:12.124708 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:12.124719 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:12.125317 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:12.126068 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:12.126101 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.127379 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.127384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:12.128901 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:12.128917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.130171 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.130184 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:12.130472 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.131101 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:12.131133 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:12.134123 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:12.138404 (+ 4281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:12.138414 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:12.138430 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:12.139037 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:12.139570 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:12.139574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:12.139577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:12.139852 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:12.140466 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:12.140474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:12.140887 (+ 413us) client_negotiation.cc:769] Sending connection context
0809 00:56:12.141147 (+ 260us) client_negotiation.cc:240] Negotiation successful
0809 00:56:12.141711 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":422,"threads_started":1}
I20260809 00:56:12.144760 1663 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35871
I20260809 00:56:12.145180 1663 heartbeater.cc:464] Registering TS with master...
I20260809 00:56:12.146250 1663 heartbeater.cc:511] Master 127.9.225.62:35871 requested a full tablet report, sending...
I20260809 00:56:12.148046 923 ts_manager.cc:194] Registered new tserver with Master: a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947)
I20260809 00:56:12.148939 923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:50431
I20260809 00:56:12.152092 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:56:12.160430 1672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.153303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.9.225.62:35871)
0809 00:56:12.153649 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:12.153656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:12.154009 (+ 353us) server_negotiation.cc:407] Connection header received
0809 00:56:12.154312 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:12.154339 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:12.154593 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:12.154789 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:56:12.155274 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.156009 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.157698 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.158192 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.158364 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.159125 (+ 761us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:56:12.159721 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:56:12.159902 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:12.160015 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 00:56:12.160116 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260809 00:56:12.161707 923 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:56:12.170300 923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:56:12.209729 1681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.185127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38923 (local address 127.0.0.1:37680)
0809 00:56:12.198730 (+ 13603us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.198746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:12.198927 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:12.200357 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:12.200371 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:12.200422 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:12.201137 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:12.201160 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.203821 (+ 2661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.203828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:12.206546 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:12.206562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.208678 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.208685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:12.208911 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.208996 (+ 85us) client_negotiation.cc:769] Sending connection context
0809 00:56:12.209109 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 00:56:12.209267 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13489,"thread_start_us":12245,"threads_started":1}
I20260809 00:56:12.210508 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.185846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37680 (local address 127.9.225.1:38923)
0809 00:56:12.197309 (+ 11463us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:12.197322 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:12.198891 (+ 1569us) server_negotiation.cc:407] Connection header received
0809 00:56:12.199147 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:12.199161 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:12.199542 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:12.199915 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:12.201501 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.203632 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.206731 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.208426 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.209502 (+ 1076us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.209826 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:12.209930 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 00:56:12.210135 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11208,"thread_start_us":10989,"threads_started":1}
I20260809 00:56:12.211295 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.185532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:35855 (local address 127.0.0.1:38114)
0809 00:56:12.202442 (+ 16910us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.202466 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:12.202608 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:12.203388 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:12.203393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:12.203420 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:12.203928 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:12.203944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.206069 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.206075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:12.208118 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:12.208133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.210188 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.210195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:12.210369 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.210405 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 00:56:12.210490 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 00:56:12.210878 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16783,"mutex_wait_us":27,"thread_start_us":16642,"threads_started":1}
I20260809 00:56:12.211561 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.185849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38114 (local address 127.9.225.2:35855)
0809 00:56:12.201951 (+ 16102us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:12.201964 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:12.202740 (+ 776us) server_negotiation.cc:407] Connection header received
0809 00:56:12.202875 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:12.202884 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:12.203157 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:12.203398 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:12.204185 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.205850 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.208329 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.209766 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.210466 (+ 700us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.210815 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:12.210905 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 00:56:12.211077 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15869,"thread_start_us":15716,"threads_started":1}
I20260809 00:56:12.212184 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.184895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:37947 (local address 127.0.0.1:60926)
0809 00:56:12.197845 (+ 12950us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.197906 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:12.198162 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:12.203462 (+ 5300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:12.203469 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:12.203525 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:12.204200 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:12.204217 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.206593 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.206602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:12.208888 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:12.208903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.210844 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.210851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:12.211034 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.211068 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 00:56:12.211143 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 00:56:12.211301 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12646,"mutex_wait_us":26,"spinlock_wait_cycles":4992,"thread_start_us":13554,"threads_started":1}
I20260809 00:56:12.213063 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.185113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.9.225.3:37947)
0809 00:56:12.202922 (+ 17809us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:12.202930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:12.202960 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:56:12.203041 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:12.203047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:12.203231 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:12.204444 (+ 1213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:12.204790 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.206382 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.209083 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.210611 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.212029 (+ 1418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.212393 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:12.212476 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 00:56:12.212657 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17629,"thread_start_us":17460,"threads_started":1}
I20260809 00:56:12.215353 1122 tablet_service.cc:1467] Processing CreateTablet for tablet ab7235a2726e4288b37ece078781cb5d (DEFAULT_TABLE table=test-table [id=8617f5ae1a0a48258100f82232433acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:56:12.215781 1350 tablet_service.cc:1467] Processing CreateTablet for tablet ab7235a2726e4288b37ece078781cb5d (DEFAULT_TABLE table=test-table [id=8617f5ae1a0a48258100f82232433acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:56:12.216878 1122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7235a2726e4288b37ece078781cb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:12.217170 1350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7235a2726e4288b37ece078781cb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:12.217244 1555 tablet_service.cc:1467] Processing CreateTablet for tablet ab7235a2726e4288b37ece078781cb5d (DEFAULT_TABLE table=test-table [id=8617f5ae1a0a48258100f82232433acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:56:12.218777 1555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7235a2726e4288b37ece078781cb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:12.230860 1694 tablet_bootstrap.cc:492] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Bootstrap starting.
I20260809 00:56:12.235409 1694 tablet_bootstrap.cc:654] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:12.235952 1698 tablet_bootstrap.cc:492] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f: Bootstrap starting.
I20260809 00:56:12.237293 1694 log.cc:824] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:12.240406 1698 tablet_bootstrap.cc:654] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:12.241444 1694 tablet_bootstrap.cc:492] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: No bootstrap required, opened a new log
I20260809 00:56:12.241801 1694 ts_tablet_manager.cc:1397] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260809 00:56:12.241951 1698 log.cc:824] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:12.242864 1695 tablet_bootstrap.cc:492] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202: Bootstrap starting.
I20260809 00:56:12.246866 1698 tablet_bootstrap.cc:492] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f: No bootstrap required, opened a new log
I20260809 00:56:12.247558 1698 ts_tablet_manager.cc:1397] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260809 00:56:12.249552 1695 tablet_bootstrap.cc:654] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:12.251089 1695 log.cc:824] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:12.253517 1694 raft_consensus.cc:348] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:12.254069 1694 raft_consensus.cc:374] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:12.254187 1694 raft_consensus.cc:729] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156cb7dab3294f11a63a1cb8e16f5afe, State: Initialized, Role: FOLLOWER
I20260809 00:56:12.255210 1695 tablet_bootstrap.cc:492] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202: No bootstrap required, opened a new log
I20260809 00:56:12.255079 1694 consensus_queue.cc:260] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:12.255563 1695 ts_tablet_manager.cc:1397] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20260809 00:56:12.256673 1698 raft_consensus.cc:348] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:12.257138 1698 raft_consensus.cc:374] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:12.257275 1698 raft_consensus.cc:729] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863d6bfc878544068e82693fdb0f1c0f, State: Initialized, Role: FOLLOWER
I20260809 00:56:12.257483 1694 ts_tablet_manager.cc:1428] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260809 00:56:12.258155 1698 consensus_queue.cc:260] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:12.259505 1448 heartbeater.cc:503] Master 127.9.225.62:35871 was elected leader, sending a full tablet report...
I20260809 00:56:12.260177 1698 ts_tablet_manager.cc:1428] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260809 00:56:12.266808 1695 raft_consensus.cc:348] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:12.267274 1695 raft_consensus.cc:374] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:12.267387 1695 raft_consensus.cc:729] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f297c7bfdf4e5db0e41af3640a4202, State: Initialized, Role: FOLLOWER
I20260809 00:56:12.268162 1695 consensus_queue.cc:260] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:12.269510 1663 heartbeater.cc:503] Master 127.9.225.62:35871 was elected leader, sending a full tablet report...
I20260809 00:56:12.270159 1695 ts_tablet_manager.cc:1428] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
W20260809 00:56:12.297170 1244 tablet.cc:2369] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:56:12.331583 1451 tablet.cc:2369] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:56:12.342422 1704 raft_consensus.cc:482] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:56:12.342737 1704 raft_consensus.cc:504] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:12.344862 1704 leader_election.cc:288] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947), 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855)
I20260809 00:56:12.355219 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.345863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:55669 (local address 127.9.225.2:35855)
0809 00:56:12.346093 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:12.346100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:12.346657 (+ 557us) server_negotiation.cc:407] Connection header received
0809 00:56:12.346865 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:12.346871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:12.346949 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:12.347050 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:12.347755 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.348919 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.352317 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.353732 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.353909 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.354009 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:12.354764 (+ 755us) server_negotiation.cc:299] Negotiation successful
0809 00:56:12.354925 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 00:56:12.355225 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.345277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:36531 (local address 127.9.225.3:37947)
0809 00:56:12.345552 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:12.345561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:12.346367 (+ 806us) server_negotiation.cc:407] Connection header received
0809 00:56:12.346630 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:12.346640 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:12.346734 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:12.346860 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:12.347755 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.348921 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.352315 (+ 3394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.353732 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.353890 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.354009 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:12.354764 (+ 755us) server_negotiation.cc:299] Negotiation successful
0809 00:56:12.354925 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260809 00:56:12.355391 1709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.345305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:37947 (local address 127.9.225.1:36531)
0809 00:56:12.346230 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.346252 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:12.346465 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:12.346885 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:12.346888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:12.346907 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:12.347534 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:12.347546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.349087 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.349093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:12.352047 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:12.352071 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.354083 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.354091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:12.354404 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.354516 (+ 112us) client_negotiation.cc:769] Sending connection context
0809 00:56:12.354627 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 00:56:12.354841 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":338,"spinlock_wait_cycles":29568,"thread_start_us":490,"threads_started":1}
I20260809 00:56:12.355696 1710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.345460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:35855 (local address 127.9.225.1:55669)
0809 00:56:12.346562 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.346578 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:12.346699 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:12.347076 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:12.347082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:12.347127 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:12.347534 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:12.347546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.349087 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.349093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:12.352048 (+ 2955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:12.352071 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.354099 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.354105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:12.354404 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.354514 (+ 110us) client_negotiation.cc:769] Sending connection context
0809 00:56:12.354627 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 00:56:12.354841 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":320,"thread_start_us":653,"threads_started":1}
I20260809 00:56:12.356842 1384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7235a2726e4288b37ece078781cb5d" candidate_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863d6bfc878544068e82693fdb0f1c0f" is_pre_election: true
I20260809 00:56:12.356842 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7235a2726e4288b37ece078781cb5d" candidate_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" is_pre_election: true
I20260809 00:56:12.357628 1587 raft_consensus.cc:2454] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156cb7dab3294f11a63a1cb8e16f5afe in term 0.
I20260809 00:56:12.357630 1384 raft_consensus.cc:2454] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 156cb7dab3294f11a63a1cb8e16f5afe in term 0.
I20260809 00:56:12.358896 1041 leader_election.cc:302] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 156cb7dab3294f11a63a1cb8e16f5afe, 863d6bfc878544068e82693fdb0f1c0f, a2f297c7bfdf4e5db0e41af3640a4202; no voters:
I20260809 00:56:12.359505 1704 raft_consensus.cc:2781] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:56:12.359700 1704 raft_consensus.cc:482] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:56:12.359860 1704 raft_consensus.cc:3037] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:12.363087 1704 raft_consensus.cc:504] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:12.363840 1704 leader_election.cc:288] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 1 election: Requested vote from peers a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947), 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855)
I20260809 00:56:12.364506 1587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7235a2726e4288b37ece078781cb5d" candidate_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f297c7bfdf4e5db0e41af3640a4202"
I20260809 00:56:12.364524 1384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7235a2726e4288b37ece078781cb5d" candidate_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863d6bfc878544068e82693fdb0f1c0f"
I20260809 00:56:12.364771 1587 raft_consensus.cc:3037] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:12.364784 1384 raft_consensus.cc:3037] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:12.367391 1587 raft_consensus.cc:2454] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 156cb7dab3294f11a63a1cb8e16f5afe in term 1.
I20260809 00:56:12.368059 1041 leader_election.cc:302] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156cb7dab3294f11a63a1cb8e16f5afe, a2f297c7bfdf4e5db0e41af3640a4202; no voters:
W20260809 00:56:12.368396 1664 tablet.cc:2369] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:56:12.368498 1704 raft_consensus.cc:2781] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:56:12.368762 1384 raft_consensus.cc:2454] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 156cb7dab3294f11a63a1cb8e16f5afe in term 1.
I20260809 00:56:12.369179 1704 raft_consensus.cc:686] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 LEADER]: Becoming Leader. State: Replica: 156cb7dab3294f11a63a1cb8e16f5afe, State: Running, Role: LEADER
I20260809 00:56:12.369815 1704 consensus_queue.cc:237] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:12.375408 923 catalog_manager.cc:5495] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe reported cstate change: term changed from 0 to 1, leader changed from <none> to 156cb7dab3294f11a63a1cb8e16f5afe (127.9.225.1). New cstate: current_term: 1 leader_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } health_report { overall_health: UNKNOWN } } }
I20260809 00:56:12.414311 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.407339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.9.225.1:38923)
0809 00:56:12.407532 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:12.407537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:12.407555 (+ 18us) server_negotiation.cc:407] Connection header received
0809 00:56:12.407690 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:12.407694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:12.407799 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:12.407937 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:56:12.408463 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.409717 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.411806 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.412244 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.412363 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.412925 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:56:12.413720 (+ 795us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:56:12.413842 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:12.414011 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 00:56:12.414093 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 00:56:12.424139 1587 raft_consensus.cc:1261] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202 [term 1 FOLLOWER]: Refusing update from remote peer 156cb7dab3294f11a63a1cb8e16f5afe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:56:12.424134 1384 raft_consensus.cc:1261] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f [term 1 FOLLOWER]: Refusing update from remote peer 156cb7dab3294f11a63a1cb8e16f5afe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:56:12.425642 1712 consensus_queue.cc:1035] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [LEADER]: Connected to new peer: Peer: permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:56:12.426167 1704 consensus_queue.cc:1035] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:56:12.445128 1716 mvcc.cc:205] Tried to move back new op lower bound from 7316426639035617280 to 7316426638831222784. Current Snapshot: MvccSnapshot[applied={T|T < 7316426639035617280}]
I20260809 00:56:12.479094 923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:56:12.479453 923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:56:12.483798 923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Sending DeleteTablet for 3 replicas of tablet ab7235a2726e4288b37ece078781cb5d
I20260809 00:56:12.485414 1350 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7235a2726e4288b37ece078781cb5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:56:12 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:38114
I20260809 00:56:12.485430 1555 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7235a2726e4288b37ece078781cb5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:56:12 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:60926
I20260809 00:56:12.486310 1122 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7235a2726e4288b37ece078781cb5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:56:12 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:37680
I20260809 00:56:12.486537 1734 tablet_replica.cc:330] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202: stopping tablet replica
I20260809 00:56:12.487076 1734 raft_consensus.cc:2229] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:56:12.487432 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 990
I20260809 00:56:12.487709 1734 raft_consensus.cc:2258] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:56:12.489754 1733 tablet_replica.cc:330] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f: stopping tablet replica
I20260809 00:56:12.490361 1733 raft_consensus.cc:2229] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:56:12.491583 1733 raft_consensus.cc:2258] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:56:12.494899 1733 ts_tablet_manager.cc:1905] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:56:12.496505 1734 ts_tablet_manager.cc:1905] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:56:12.500916 1735 tablet_replica.cc:330] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: stopping tablet replica
I20260809 00:56:12.501899 1735 raft_consensus.cc:2229] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:56:12.503173 1735 raft_consensus.cc:2258] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:56:12.506624 1733 ts_tablet_manager.cc:1918] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:56:12.507028 1733 log.cc:1196] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/ab7235a2726e4288b37ece078781cb5d
I20260809 00:56:12.507956 1733 ts_tablet_manager.cc:1939] T ab7235a2726e4288b37ece078781cb5d P 863d6bfc878544068e82693fdb0f1c0f: Deleting consensus metadata
I20260809 00:56:12.511241 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.488287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37694 (local address 127.9.225.1:38923)
0809 00:56:12.488475 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:12.488481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:12.488663 (+ 182us) server_negotiation.cc:407] Connection header received
0809 00:56:12.488900 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:12.488906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:12.488992 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:12.489108 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:12.491079 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.492369 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.494210 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.494927 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.495075 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.499261 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:12.499305 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:12.499343 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:12.499436 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:12.504756 (+ 5320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:12.506390 (+ 1634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:12.506404 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:12.506409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:12.506668 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:12.507336 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:12.507341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:12.507344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:12.508055 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:12.508535 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:12.510332 (+ 1797us) server_negotiation.cc:299] Negotiation successful
0809 00:56:12.510736 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 00:56:12.511240 898 catalog_manager.cc:4841] TS 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855): tablet ab7235a2726e4288b37ece078781cb5d (table test-table [id=8617f5ae1a0a48258100f82232433acc]) successfully deleted
I20260809 00:56:12.512624 1231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:12.513633 1734 ts_tablet_manager.cc:1918] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:56:12.513993 1734 log.cc:1196] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/ab7235a2726e4288b37ece078781cb5d
I20260809 00:56:12.514933 1734 ts_tablet_manager.cc:1939] T ab7235a2726e4288b37ece078781cb5d P a2f297c7bfdf4e5db0e41af3640a4202: Deleting consensus metadata
I20260809 00:56:12.518271 895 catalog_manager.cc:4841] TS a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947): tablet ab7235a2726e4288b37ece078781cb5d (table test-table [id=8617f5ae1a0a48258100f82232433acc]) successfully deleted
I20260809 00:56:12.513491 1735 ts_tablet_manager.cc:1905] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:56:12.621512 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 990
W20260809 00:56:12.664179 897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.225.1:38923 (error 108)
I20260809 00:56:12.665295 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 1247
I20260809 00:56:12.666833 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.666141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38923 (local address 127.0.0.1:37698)
0809 00:56:12.666440 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.666598 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260809 00:56:12.667176 897 catalog_manager.cc:4584] TS 156cb7dab3294f11a63a1cb8e16f5afe (127.9.225.1:38923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab7235a2726e4288b37ece078781cb5d: Network error: Client connection negotiation failed: client connection to 127.9.225.1:38923: connect: Connection refused (error 111)
I20260809 00:56:12.685369 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.666552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.9.225.2:35855)
0809 00:56:12.666824 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:12.666833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:12.666863 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:56:12.667234 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:12.667241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:12.667379 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:12.667525 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:12.669595 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.670448 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.672196 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.672621 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.672739 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.676080 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:12.676102 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:12.676123 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:12.676176 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:12.679879 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:12.681216 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:12.681226 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:12.681228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:12.681683 (+ 455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:12.682550 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:12.682556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:12.682559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:12.683573 (+ 1014us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:12.683955 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:12.684518 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 00:56:12.684860 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260809 00:56:12.686225 1441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:12.721946 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.721005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38923 (local address 127.0.0.1:37702)
0809 00:56:12.721525 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.721707 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260809 00:56:12.723658 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.723146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38923 (local address 127.0.0.1:37712)
0809 00:56:12.723361 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.723499 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 00:56:12.784407 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.783845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38923 (local address 127.0.0.1:37714)
0809 00:56:12.784103 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.784212 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 00:56:12.785648 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.785246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38923 (local address 127.0.0.1:37724)
0809 00:56:12.785456 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.785530 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 00:56:12.788836 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 1247
I20260809 00:56:12.819065 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 1455
I20260809 00:56:12.835834 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.819978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60930 (local address 127.9.225.3:37947)
0809 00:56:12.820184 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:12.820195 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:12.820221 (+ 26us) server_negotiation.cc:407] Connection header received
0809 00:56:12.820310 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:12.820314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:12.820399 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:12.820493 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:12.822169 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.822941 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.824495 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:12.824927 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:12.825036 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:12.829162 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:12.829179 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:12.829200 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:12.829248 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:12.832530 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:12.833659 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:12.833669 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:12.833671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:12.833852 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:12.834306 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:12.834310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:12.834312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:12.834761 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:12.835012 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:12.835335 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 00:56:12.835554 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 00:56:12.836629 1656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:12.866593 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.865974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38923 (local address 127.0.0.1:37738)
0809 00:56:12.866247 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.866354 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260809 00:56:12.867945 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:12.867487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38923 (local address 127.0.0.1:37752)
0809 00:56:12.867713 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:12.867817 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260809 00:56:12.932212 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 1455
I20260809 00:56:12.960901 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 871
I20260809 00:56:12.962347 977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:13.084407 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:13.083659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38923 (local address 127.0.0.1:37762)
0809 00:56:13.084081 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:13.084195 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:38923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244}
I20260809 00:56:13.084452 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 871
I20260809 00:56:13.120585 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35871
--webserver_interface=127.9.225.62
--webserver_port=43677
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:35871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:56:13.509073 1743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:13.509466 1743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:13.509641 1743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:13.517745 1743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:56:13.517902 1743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:13.517952 1743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:13.518054 1743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:56:13.518126 1743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:56:13.529858 1743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:35871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=43677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:13.531490 1743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:13.533003 1743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:13.546465 1757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:13.546630 1760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:13.546808 1755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:13.547909 1743 server_base.cc:1034] running on GCE node
I20260809 00:56:13.548880 1743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:13.550702 1743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:13.552022 1743 hybrid_clock.cc:648] HybridClock initialized: now 1786236973551976 us; error 56 us; skew 500 ppm
I20260809 00:56:13.552484 1743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:13 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236973, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:13.556280 1743 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:56:13.558318 1743 webserver.cc:460] Webserver started at http://127.9.225.62:43677/ using document root <none> and password file <none>
I20260809 00:56:13.559125 1743 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:13.559253 1743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:13.565897 1743 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:56:13.571926 1769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:13.573750 1743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:56:13.574009 1743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "25d7d397d6734f85a2f483ee67352746"
format_stamp: "Formatted at 2026-08-09 00:56:06 on dist-test-slave-j2vl"
server_key: "2ddd14e9279df414455ac81771e4af87"
server_key_iv: "0f81b8ba5af581ba62706611f8a53739"
server_key_version: "encryptionkey@0"
I20260809 00:56:13.575160 1743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:13.606000 1743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:13.611126 1743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:13.611413 1743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:13.639401 1743 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:35871
I20260809 00:56:13.639421 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:35871 every 8 connection(s)
I20260809 00:56:13.641672 1743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:56:13.646503 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 1743
I20260809 00:56:13.647400 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:38923
--local_ip_for_outbound_sockets=127.9.225.1
--tserver_master_addrs=127.9.225.62:35871
--webserver_port=44941
--webserver_interface=127.9.225.1
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:56:13.649793 1855 sys_catalog.cc:263] Verifying existing consensus state
I20260809 00:56:13.653599 1855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Bootstrap starting.
I20260809 00:56:13.676486 1855 log.cc:824] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:13.696599 1855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 00:56:13.697299 1855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746: Bootstrap complete.
I20260809 00:56:13.707350 1855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } }
I20260809 00:56:13.708424 1855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d7d397d6734f85a2f483ee67352746, State: Initialized, Role: FOLLOWER
I20260809 00:56:13.709132 1855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } }
I20260809 00:56:13.709453 1855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:56:13.709605 1855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:56:13.709813 1855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 2 FOLLOWER]: Advancing to term 3
I20260809 00:56:13.714578 1855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } }
I20260809 00:56:13.715337 1855 leader_election.cc:302] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25d7d397d6734f85a2f483ee67352746; no voters:
I20260809 00:56:13.716396 1855 leader_election.cc:288] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 00:56:13.716565 1866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 00:56:13.717967 1866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [term 3 LEADER]: Becoming Leader. State: Replica: 25d7d397d6734f85a2f483ee67352746, State: Running, Role: LEADER
I20260809 00:56:13.718744 1866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } }
I20260809 00:56:13.719152 1855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:56:13.721894 1870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25d7d397d6734f85a2f483ee67352746. Latest consensus state: current_term: 3 leader_uuid: "25d7d397d6734f85a2f483ee67352746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } } }
I20260809 00:56:13.722368 1870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:13.722298 1867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "25d7d397d6734f85a2f483ee67352746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d7d397d6734f85a2f483ee67352746" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35871 } } }
I20260809 00:56:13.722735 1867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d7d397d6734f85a2f483ee67352746 [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:13.731536 1878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:56:13.742082 1878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:56:13.744097 1878 catalog_manager.cc:1259] Loaded cluster ID: ee8a326af34041dda221f75293b6c567
I20260809 00:56:13.744303 1878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 00:56:13.746819 1878 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0c394de9ad google::LogMessage::Fail() at ??:0
@ 0x7f0c394e188a google::LogMessage::SendToLog() at ??:0
@ 0x7f0c394de4de google::LogMessage::Flush() at ??:0
@ 0x7f0c394dfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0c4ef11512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0c4ef0e976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0c4ef96610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0c4ef96439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0c3c12a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0c3c12d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0c3c12d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0c3c108eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0c3f3e26db start_thread at ??:0
@ 0x7f0c3837861f clone at ??:0
W20260809 00:56:14.121848 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:14.122184 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:14.122373 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:14.130513 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:14.130678 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:14.130895 1858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:56:14.142589 1858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:38923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=44941
--tserver_master_addrs=127.9.225.62:35871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:14.144163 1858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:14.145709 1858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:14.160595 1931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:14.160764 1929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:14.160969 1927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:14.161988 1858 server_base.cc:1034] running on GCE node
I20260809 00:56:14.162732 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:14.163836 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:14.165074 1858 hybrid_clock.cc:648] HybridClock initialized: now 1786236974165018 us; error 41 us; skew 500 ppm
I20260809 00:56:14.165536 1858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:14 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236974, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:14.169466 1858 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:56:14.171303 1858 webserver.cc:460] Webserver started at http://127.9.225.1:44941/ using document root <none> and password file <none>
I20260809 00:56:14.172120 1858 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:14.172289 1858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:14.178797 1858 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 00:56:14.182475 1946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:14.184327 1858 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260809 00:56:14.184617 1858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "156cb7dab3294f11a63a1cb8e16f5afe"
format_stamp: "Formatted at 2026-08-09 00:56:07 on dist-test-slave-j2vl"
server_key: "0dcbe8098e776157e256bf5babda35e6"
server_key_iv: "d78ac24cb8d98107e95f64bad47f43b4"
server_key_version: "encryptionkey@0"
I20260809 00:56:14.185877 1858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:14.211452 1858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:14.217926 1858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:14.218312 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:14.219568 1858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:14.221695 1955 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260809 00:56:14.227929 1858 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 00:56:14.228083 1858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260809 00:56:14.228227 1858 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 00:56:14.231489 1858 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 00:56:14.231606 1858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260809 00:56:14.231993 1955 tablet_bootstrap.cc:492] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Bootstrap starting.
I20260809 00:56:14.243544 1955 log.cc:824] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:14.263887 1955 tablet_bootstrap.cc:492] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 00:56:14.264649 1955 tablet_bootstrap.cc:492] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Bootstrap complete.
I20260809 00:56:14.265240 1955 ts_tablet_manager.cc:1397] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.013s
I20260809 00:56:14.277877 1955 raft_consensus.cc:348] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:14.279543 1955 raft_consensus.cc:729] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156cb7dab3294f11a63a1cb8e16f5afe, State: Initialized, Role: FOLLOWER
I20260809 00:56:14.280434 1955 consensus_queue.cc:260] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:14.281222 1858 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:38923
I20260809 00:56:14.281309 2125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:38923 every 8 connection(s)
I20260809 00:56:14.283027 1955 ts_tablet_manager.cc:1428] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe: Time spent starting tablet: real 0.018s user 0.006s sys 0.014s
I20260809 00:56:14.283195 1858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:56:14.286129 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 1858
I20260809 00:56:14.286850 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:35855
--local_ip_for_outbound_sockets=127.9.225.2
--tserver_master_addrs=127.9.225.62:35871
--webserver_port=41007
--webserver_interface=127.9.225.2
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:56:14.288736 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:14.286356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:39277)
0809 00:56:14.287377 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:14.287682 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":443,"threads_started":1}
W20260809 00:56:14.289701 2127 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:35871 (0 consecutive failures): Network error: Failed to ping master at 127.9.225.62:35871: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
I20260809 00:56:14.291044 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:14.290547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:40543)
0809 00:56:14.290731 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:14.290809 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 00:56:14.292085 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:14.291685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:48757)
0809 00:56:14.291853 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:14.291952 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 00:56:14.292336 2127 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 00:56:14.678716 2135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:14.679109 2135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:14.679220 2135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:14.686952 2135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:14.687106 2135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:14.687232 2135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:56:14.699226 2135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:35855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=41007
--tserver_master_addrs=127.9.225.62:35871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:14.700706 2135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:14.702250 2135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:14.717913 2144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:14.717988 2145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:14.717851 2148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:14.718716 2135 server_base.cc:1034] running on GCE node
I20260809 00:56:14.719681 2135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:14.720892 2135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:14.722172 2135 hybrid_clock.cc:648] HybridClock initialized: now 1786236974722120 us; error 52 us; skew 500 ppm
I20260809 00:56:14.722587 2135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:14 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236974, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:14.727610 2135 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:56:14.729569 2135 webserver.cc:460] Webserver started at http://127.9.225.2:41007/ using document root <none> and password file <none>
I20260809 00:56:14.730473 2135 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:14.730649 2135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:14.737051 2135 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 00:56:14.740402 2159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:14.742115 2135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 00:56:14.742365 2135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "863d6bfc878544068e82693fdb0f1c0f"
format_stamp: "Formatted at 2026-08-09 00:56:08 on dist-test-slave-j2vl"
server_key: "4c07c22e534eaf6c5d28e5e8ef1d01da"
server_key_iv: "aa0986f4e363b3c884e5adcab7623a7f"
server_key_version: "encryptionkey@0"
I20260809 00:56:14.743559 2135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:14.773763 2135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:14.778865 2135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:14.779239 2135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:14.780546 2135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:14.782208 2135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:14.782366 2135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:14.782517 2135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:14.782630 2135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:14.820932 2135 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:35855
I20260809 00:56:14.820986 2358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:35855 every 8 connection(s)
I20260809 00:56:14.822600 2135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:56:14.824321 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 2135
I20260809 00:56:14.825073 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:37947
--local_ip_for_outbound_sockets=127.9.225.3
--tserver_master_addrs=127.9.225.62:35871
--webserver_port=41689
--webserver_interface=127.9.225.3
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:56:14.829183 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:14.826456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:33347)
0809 00:56:14.827498 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:14.827800 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":378,"threads_started":1}
W20260809 00:56:14.830243 2359 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:35871 (0 consecutive failures): Network error: Failed to ping master at 127.9.225.62:35871: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
I20260809 00:56:14.831485 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:14.830944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:60291)
0809 00:56:14.831176 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:14.831302 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 00:56:14.832634 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:14.832151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:43171)
0809 00:56:14.832350 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:14.832459 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 00:56:14.832975 2359 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 00:56:15.223429 2366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:15.223951 2366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:15.224149 2366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:15.237154 2366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:15.237370 2366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:15.237896 2366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:56:15.255240 2366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:37947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=41689
--tserver_master_addrs=127.9.225.62:35871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:15.257654 2366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:15.259953 2366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:15.278396 2373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:15.278396 2374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:15.278445 2376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:15.279718 2366 server_base.cc:1034] running on GCE node
I20260809 00:56:15.280622 2366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:15.282076 2366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:15.283370 2366 hybrid_clock.cc:648] HybridClock initialized: now 1786236975283312 us; error 51 us; skew 500 ppm
I20260809 00:56:15.283813 2366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:15 dist-test-slave-j2vl krb5kdc[32452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236975, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:15.288347 2366 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:56:15.291724 2366 webserver.cc:460] Webserver started at http://127.9.225.3:41689/ using document root <none> and password file <none>
I20260809 00:56:15.292716 2366 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:15.292937 2366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:15.294034 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:15.293037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:53301)
0809 00:56:15.293655 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:15.293797 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":387,"threads_started":1}
I20260809 00:56:15.299742 2366 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:56:15.303504 2390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:15.305757 2366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260809 00:56:15.306031 2366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "a2f297c7bfdf4e5db0e41af3640a4202"
format_stamp: "Formatted at 2026-08-09 00:56:08 on dist-test-slave-j2vl"
server_key: "b1a2c929fcdf62a8981ffb444189b006"
server_key_iv: "a512616f7af1e55c723841fb3218a3d7"
server_key_version: "encryptionkey@0"
I20260809 00:56:15.307281 2366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:15.342971 2366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:15.349036 2366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:15.349444 2366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:15.350804 2366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:15.352499 2366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:15.352658 2366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:15.352802 2366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:15.352895 2366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:15.392817 2366 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:37947
I20260809 00:56:15.392923 2564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:37947 every 8 connection(s)
I20260809 00:56:15.394594 2366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:56:15.395562 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 2366
I20260809 00:56:15.401288 2570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:15.398937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:41813)
0809 00:56:15.399906 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:15.400157 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":353,"threads_started":1}
W20260809 00:56:15.402154 2567 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:35871 (0 consecutive failures): Network error: Failed to ping master at 127.9.225.62:35871: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
I20260809 00:56:15.403160 2570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:15.402722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:51245)
0809 00:56:15.402954 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:15.403036 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260809 00:56:15.404421 2570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:15.404031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:36541)
0809 00:56:15.404182 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:15.404269 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 00:56:15.404757 2567 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 00:56:15.794061 2571 raft_consensus.cc:482] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:56:15.794449 2571 raft_consensus.cc:504] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:15.796255 2571 leader_election.cc:288] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947), 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855)
Aug 09 00:56:15 dist-test-slave-j2vl krb5kdc[32452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236974, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
Aug 09 00:56:15 dist-test-slave-j2vl krb5kdc[32452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236974, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:56:15.834142 2577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:15.796933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:50963 (local address 127.9.225.3:37947)
0809 00:56:15.808731 (+ 11798us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:15.808747 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:15.809107 (+ 360us) server_negotiation.cc:407] Connection header received
0809 00:56:15.811392 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:15.811402 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:15.811966 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:15.812367 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:15.814978 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:15.816993 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:15.819484 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:15.820567 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:15.820778 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:15.825883 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:15.825899 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:15.825920 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:15.825987 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:15.829734 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:15.830807 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:15.830816 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:15.830822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:15.831018 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:15.831540 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:15.831544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:15.831546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:15.832129 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:15.832351 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:15.833267 (+ 916us) server_negotiation.cc:299] Negotiation successful
0809 00:56:15.833729 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11597,"thread_start_us":11461,"threads_started":1}
I20260809 00:56:15.834133 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:15.796777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:35855 (local address 127.9.225.1:37365)
0809 00:56:15.809116 (+ 12339us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:15.809132 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:15.809526 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:15.812852 (+ 3326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:15.812858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:15.813615 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:15.814644 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:15.814671 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:15.817284 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:15.817289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:15.819299 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:15.819318 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:15.820814 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:15.820822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:15.821188 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:15.822173 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:15.822216 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:15.825411 (+ 3195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:15.829971 (+ 4560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:15.829975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:15.829978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:15.830626 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:15.831141 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:15.831148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:15.831154 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:15.831392 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:15.832285 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:15.832294 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:15.832681 (+ 387us) client_negotiation.cc:769] Sending connection context
0809 00:56:15.832985 (+ 304us) client_negotiation.cc:240] Negotiation successful
0809 00:56:15.833628 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12202,"mutex_wait_us":214,"spinlock_wait_cycles":90752,"thread_start_us":11998,"threads_started":1}
I20260809 00:56:15.834383 2576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:15.796994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:37365 (local address 127.9.225.2:35855)
0809 00:56:15.808155 (+ 11161us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:15.808184 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:15.809268 (+ 1084us) server_negotiation.cc:407] Connection header received
0809 00:56:15.811407 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:15.811415 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:15.812056 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:15.812379 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:15.814936 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:15.816993 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:15.819539 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:15.820542 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:15.820792 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:15.825812 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:15.825824 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:15.825845 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:15.825901 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:15.829771 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:15.830810 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:15.830818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:15.830823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:15.831005 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:15.831530 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:15.831536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:15.831546 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:15.832126 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:15.832361 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:15.833238 (+ 877us) server_negotiation.cc:299] Negotiation successful
0809 00:56:15.833745 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10854,"thread_start_us":10658,"threads_started":1}
I20260809 00:56:15.835565 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:15.796580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:37947 (local address 127.9.225.1:50963)
0809 00:56:15.808673 (+ 12093us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:15.808748 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:15.809509 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:15.812547 (+ 3038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:15.812559 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:15.813616 (+ 1057us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:15.814641 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:15.814670 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:15.817290 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:15.817294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:15.819243 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:15.819266 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:15.821129 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:15.821137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:15.821396 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:15.822490 (+ 1094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:15.822513 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:15.825484 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:15.829943 (+ 4459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:15.829956 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:15.829976 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:15.830618 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:15.831179 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:15.831183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:15.831186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:15.831401 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:15.832285 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:15.832293 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:15.832680 (+ 387us) client_negotiation.cc:769] Sending connection context
0809 00:56:15.832985 (+ 305us) client_negotiation.cc:240] Negotiation successful
0809 00:56:15.835275 (+ 2290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11863,"mutex_wait_us":81,"spinlock_wait_cycles":17024,"thread_start_us":12181,"threads_started":1}
I20260809 00:56:15.835597 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:15.833616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:36079)
0809 00:56:15.835019 (+ 1403us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:15.835226 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1273,"thread_start_us":1120,"threads_started":1}
W20260809 00:56:15.838958 1947 leader_election.cc:341] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947): Not found: Tablet not found: ab7235a2726e4288b37ece078781cb5d
W20260809 00:56:15.839313 1950 leader_election.cc:341] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855): Not found: Tablet not found: ab7235a2726e4288b37ece078781cb5d
I20260809 00:56:15.839509 1950 leader_election.cc:302] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 156cb7dab3294f11a63a1cb8e16f5afe; no voters: 863d6bfc878544068e82693fdb0f1c0f, a2f297c7bfdf4e5db0e41af3640a4202
I20260809 00:56:15.840021 2571 raft_consensus.cc:2726] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 00:56:16.295641 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:16.295125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:43327)
0809 00:56:16.295354 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:16.295480 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260809 00:56:16.406417 2579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:16.405553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:33713)
0809 00:56:16.406066 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:16.406183 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":243,"threads_started":1}
I20260809 00:56:16.838328 2581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:16.837434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:45097)
0809 00:56:16.838020 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:16.838134 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":306,"threads_started":1}
I20260809 00:56:17.297897 2582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:17.296834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:34423)
0809 00:56:17.297515 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:17.297633 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":289,"threads_started":1}
I20260809 00:56:17.408519 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:17.407597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:58867)
0809 00:56:17.408190 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:17.408298 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":346,"threads_started":1}
I20260809 00:56:17.759130 2584 raft_consensus.cc:482] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:56:17.759395 2584 raft_consensus.cc:504] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:17.760175 2584 leader_election.cc:288] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947), 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855)
W20260809 00:56:17.761444 1950 leader_election.cc:341] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855): Not found: Tablet not found: ab7235a2726e4288b37ece078781cb5d
W20260809 00:56:17.761662 1947 leader_election.cc:341] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947): Not found: Tablet not found: ab7235a2726e4288b37ece078781cb5d
I20260809 00:56:17.761794 1947 leader_election.cc:302] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 156cb7dab3294f11a63a1cb8e16f5afe; no voters: 863d6bfc878544068e82693fdb0f1c0f, a2f297c7bfdf4e5db0e41af3640a4202
I20260809 00:56:17.762058 2584 raft_consensus.cc:2726] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 00:56:17.840251 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:17.839400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:53305)
0809 00:56:17.839975 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:17.840070 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":289,"threads_started":1}
I20260809 00:56:18.299680 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:18.298870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:44081)
0809 00:56:18.299377 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:18.299478 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":277,"threads_started":1}
I20260809 00:56:18.410521 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:18.409732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:50991)
0809 00:56:18.410244 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:18.410345 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":260,"threads_started":1}
I20260809 00:56:18.842394 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:18.841481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:48871)
0809 00:56:18.842098 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:18.842193 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":355,"threads_started":1}
I20260809 00:56:19.301863 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:19.300863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:44081)
0809 00:56:19.301542 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:19.301634 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":265,"threads_started":1}
I20260809 00:56:19.413738 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:19.412360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:36679)
0809 00:56:19.413166 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:19.413312 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":373,"threads_started":1}
I20260809 00:56:19.844288 2595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:19.843531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:41025)
0809 00:56:19.844027 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:19.844116 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":253,"threads_started":1}
I20260809 00:56:20.303577 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:20.302855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:47735)
0809 00:56:20.303305 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:20.303394 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":238,"threads_started":1}
I20260809 00:56:20.415768 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:20.415008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:38761)
0809 00:56:20.415507 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:20.415595 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":243,"threads_started":1}
I20260809 00:56:20.846328 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:20.845478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:53155)
0809 00:56:20.846031 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:20.846130 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":274,"threads_started":1}
I20260809 00:56:20.976187 2602 raft_consensus.cc:482] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:56:20.976639 2602 raft_consensus.cc:504] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:20.977604 2602 leader_election.cc:288] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947), 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855)
W20260809 00:56:20.978681 1947 leader_election.cc:341] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947): Not found: Tablet not found: ab7235a2726e4288b37ece078781cb5d
W20260809 00:56:20.978896 1950 leader_election.cc:341] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855): Not found: Tablet not found: ab7235a2726e4288b37ece078781cb5d
I20260809 00:56:20.979074 1950 leader_election.cc:302] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 156cb7dab3294f11a63a1cb8e16f5afe; no voters: 863d6bfc878544068e82693fdb0f1c0f, a2f297c7bfdf4e5db0e41af3640a4202
I20260809 00:56:20.979374 2602 raft_consensus.cc:2726] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 00:56:21.305734 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:21.304683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:33717)
0809 00:56:21.305293 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:21.305492 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":317,"threads_started":1}
I20260809 00:56:21.417716 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:21.416811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:49617)
0809 00:56:21.417415 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:21.417519 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":290,"threads_started":1}
I20260809 00:56:21.848783 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:21.847605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:58227)
0809 00:56:21.848396 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:21.848509 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":331,"threads_started":1}
I20260809 00:56:22.307751 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:22.306870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:43367)
0809 00:56:22.307474 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:22.307582 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":296,"threads_started":1}
I20260809 00:56:22.419845 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:22.418902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:47655)
0809 00:56:22.419526 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:22.419621 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":296,"threads_started":1}
I20260809 00:56:22.850901 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:22.850069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:57141)
0809 00:56:22.850574 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:22.850684 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":248,"threads_started":1}
I20260809 00:56:23.263790 2614 raft_consensus.cc:482] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:56:23.264010 2614 raft_consensus.cc:504] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:23.264768 2614 leader_election.cc:288] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947), 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855)
W20260809 00:56:23.265987 1950 leader_election.cc:341] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855): Not found: Tablet not found: ab7235a2726e4288b37ece078781cb5d
W20260809 00:56:23.266242 1947 leader_election.cc:341] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947): Not found: Tablet not found: ab7235a2726e4288b37ece078781cb5d
I20260809 00:56:23.266372 1947 leader_election.cc:302] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 156cb7dab3294f11a63a1cb8e16f5afe; no voters: 863d6bfc878544068e82693fdb0f1c0f, a2f297c7bfdf4e5db0e41af3640a4202
I20260809 00:56:23.266669 2614 raft_consensus.cc:2726] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 00:56:23.309878 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:23.308871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:42481)
0809 00:56:23.309526 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:23.309661 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":278,"threads_started":1}
I20260809 00:56:23.421950 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:23.421094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:50637)
0809 00:56:23.421651 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:23.421756 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":331,"threads_started":1}
I20260809 00:56:23.852947 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:23.852019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:56913)
0809 00:56:23.852582 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:23.852688 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":276,"threads_started":1}
I20260809 00:56:24.311990 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:24.311168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:43733)
0809 00:56:24.311658 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:24.311776 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":259,"threads_started":1}
I20260809 00:56:24.423897 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:24.422990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:59581)
0809 00:56:24.423588 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:24.423699 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":298,"threads_started":1}
I20260809 00:56:24.855028 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:24.854113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:55057)
0809 00:56:24.854646 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:24.854763 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":287,"threads_started":1}
I20260809 00:56:25.314121 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:25.313052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:60857)
0809 00:56:25.313664 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:25.313804 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":322,"threads_started":1}
I20260809 00:56:25.426028 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:25.425120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:57581)
0809 00:56:25.425715 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:25.425817 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":341,"threads_started":1}
I20260809 00:56:25.857272 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:25.856194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:33999)
0809 00:56:25.856971 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:25.857080 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":386,"threads_started":1}
I20260809 00:56:26.316556 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:26.315516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:36101)
0809 00:56:26.316157 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:26.316287 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":362,"threads_started":1}
I20260809 00:56:26.428099 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:26.427245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:52637)
0809 00:56:26.427797 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:26.427918 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":277,"threads_started":1}
I20260809 00:56:26.859722 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:26.858640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:54449)
0809 00:56:26.859327 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:26.859459 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":366,"threads_started":1}
I20260809 00:56:27.318600 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:27.317704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:42647)
0809 00:56:27.318303 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:27.318389 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":358,"threads_started":1}
I20260809 00:56:27.430259 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:27.429334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:52067)
0809 00:56:27.429947 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:27.430064 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":251,"threads_started":1}
I20260809 00:56:27.861835 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:27.860919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:50367)
0809 00:56:27.861494 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:27.861615 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":252,"threads_started":1}
I20260809 00:56:28.320669 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:28.319742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:39511)
0809 00:56:28.320345 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:28.320441 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":355,"threads_started":1}
I20260809 00:56:28.432364 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:28.431522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:52993)
0809 00:56:28.432046 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:28.432152 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":270,"threads_started":1}
I20260809 00:56:28.863839 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:28.862997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:53337)
0809 00:56:28.863530 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:28.863618 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":278,"threads_started":1}
I20260809 00:56:28.954008 2642 raft_consensus.cc:482] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:56:28.954294 2642 raft_consensus.cc:504] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f297c7bfdf4e5db0e41af3640a4202" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 37947 } } peers { permanent_uuid: "156cb7dab3294f11a63a1cb8e16f5afe" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38923 } } peers { permanent_uuid: "863d6bfc878544068e82693fdb0f1c0f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 35855 } }
I20260809 00:56:28.955330 2642 leader_election.cc:288] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947), 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855)
W20260809 00:56:28.956208 1947 leader_election.cc:341] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a2f297c7bfdf4e5db0e41af3640a4202 (127.9.225.3:37947): Not found: Tablet not found: ab7235a2726e4288b37ece078781cb5d
W20260809 00:56:28.956424 1950 leader_election.cc:341] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 863d6bfc878544068e82693fdb0f1c0f (127.9.225.2:35855): Not found: Tablet not found: ab7235a2726e4288b37ece078781cb5d
I20260809 00:56:28.956594 1950 leader_election.cc:302] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 156cb7dab3294f11a63a1cb8e16f5afe; no voters: 863d6bfc878544068e82693fdb0f1c0f, a2f297c7bfdf4e5db0e41af3640a4202
I20260809 00:56:28.956878 2642 raft_consensus.cc:2726] T ab7235a2726e4288b37ece078781cb5d P 156cb7dab3294f11a63a1cb8e16f5afe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 00:56:29.323617 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:29.322337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:35499)
0809 00:56:29.323128 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:29.323261 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":353,"threads_started":1}
I20260809 00:56:29.434526 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:29.433650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:56197)
0809 00:56:29.434191 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:29.434295 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":287,"threads_started":1}
I20260809 00:56:29.866076 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:29.865229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.2:53671)
0809 00:56:29.865784 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:29.865899 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":308,"threads_started":1}
I20260809 00:56:30.325724 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:30.324803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.1:52735)
0809 00:56:30.325440 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:30.325546 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":286,"threads_started":1}
I20260809 00:56:30.402803 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 1858
I20260809 00:56:30.425217 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:30.404263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.9.225.1:38923)
0809 00:56:30.404960 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:30.404973 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:30.405014 (+ 41us) server_negotiation.cc:407] Connection header received
0809 00:56:30.405104 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:30.405115 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:30.405484 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:30.405672 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:30.407309 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:30.408382 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:30.409863 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:30.410544 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:30.410707 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:30.418417 (+ 7710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:30.418440 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:30.418459 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:30.418508 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:30.421765 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:30.422722 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:30.422730 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:30.422733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:30.422910 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:30.423352 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:30.423357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:30.423359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:30.423767 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:30.424128 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:30.424519 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 00:56:30.424839 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":285,"threads_started":1}
I20260809 00:56:30.426710 2121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:30.436997 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:30.435920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35871 (local address 127.9.225.3:48965)
0809 00:56:30.436586 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:30.436732 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:35871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":305,"threads_started":1}
I20260809 00:56:30.527060 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 1858
I20260809 00:56:30.561563 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 2135
I20260809 00:56:30.577404 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:30.562526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.9.225.2:35855)
0809 00:56:30.563199 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:30.563204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:30.563224 (+ 20us) server_negotiation.cc:407] Connection header received
0809 00:56:30.563288 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:30.563292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:30.563359 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:30.563481 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:30.565043 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:30.566108 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:30.567837 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:30.568319 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:30.568453 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:30.571435 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:30.571441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:30.571445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:30.571499 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:30.574437 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:30.575265 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:30.575269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:30.575271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:30.575431 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:30.575794 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:30.575798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:30.575800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:30.576122 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:30.576335 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:30.576683 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 00:56:30.576910 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":433,"threads_started":1}
I20260809 00:56:30.578285 2346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:30.667366 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 2135
I20260809 00:56:30.701290 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 2366
I20260809 00:56:30.716179 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:30.702408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.9.225.3:37947)
0809 00:56:30.703071 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:30.703076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:30.703095 (+ 19us) server_negotiation.cc:407] Connection header received
0809 00:56:30.703153 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:30.703156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:30.703214 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:30.703313 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:30.704834 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:30.705834 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:30.707127 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:30.707580 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:30.707680 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:30.710440 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:30.710445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:30.710449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:30.710490 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:30.713594 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:30.714320 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:30.714324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:30.714326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:30.714463 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:30.714807 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:30.714810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:30.714812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:30.715118 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:30.715354 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:30.715584 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 00:56:30.715761 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":365,"threads_started":1}
I20260809 00:56:30.717020 2558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:30.806430 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 2366
2026-08-09T00:56:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (26121 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:56:30 dist-test-slave-j2vl krb5kdc[2665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:56:30 dist-test-slave-j2vl krb5kdc[2665](info): set up 2 sockets
Aug 09 00:56:30 dist-test-slave-j2vl krb5kdc[2665](info): commencing operation
krb5kdc: starting...
W20260809 00:56:32.191440 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.331s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:56:32 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:56:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:56:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:56:32.538463 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39741
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:56:32.933230 2682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:32.933703 2682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:32.933810 2682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:32.941735 2682 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 00:56:32.941862 2682 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 00:56:32.941905 2682 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 00:56:32.941952 2682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:56:32.942013 2682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:32.942087 2682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:32.942135 2682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:56:32.942185 2682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:56:32.953971 2682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:32.955461 2682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:32.956941 2682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:32.971432 2689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:32.971370 2690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:32.971468 2694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:32.971944 2682 server_base.cc:1034] running on GCE node
I20260809 00:56:32.972999 2682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:32.974980 2682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:32.976271 2682 hybrid_clock.cc:648] HybridClock initialized: now 1786236992976164 us; error 72 us; skew 500 ppm
I20260809 00:56:32.976755 2682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:32 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236992, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:32.980986 2682 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:56:32.982937 2682 webserver.cc:460] Webserver started at http://127.9.225.62:43759/ using document root <none> and password file <none>
I20260809 00:56:32.983824 2682 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:32.983955 2682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:32.984316 2682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:56:32.987866 2682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "3c6cac20e9c347b3a5c86bf76de4e24f"
format_stamp: "Formatted at 2026-08-09 00:56:32 on dist-test-slave-j2vl"
server_key: "c86c6c96f7ebd358bda03ec42ea7c85a"
server_key_iv: "98aa85a0430e6757e83f02dacd5aeb84"
server_key_version: "encryptionkey@0"
I20260809 00:56:32.988847 2682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "3c6cac20e9c347b3a5c86bf76de4e24f"
format_stamp: "Formatted at 2026-08-09 00:56:32 on dist-test-slave-j2vl"
server_key: "c86c6c96f7ebd358bda03ec42ea7c85a"
server_key_iv: "98aa85a0430e6757e83f02dacd5aeb84"
server_key_version: "encryptionkey@0"
I20260809 00:56:32.995502 2682 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20260809 00:56:32.999953 2706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:33.001847 2682 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 00:56:33.002099 2682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "3c6cac20e9c347b3a5c86bf76de4e24f"
format_stamp: "Formatted at 2026-08-09 00:56:32 on dist-test-slave-j2vl"
server_key: "c86c6c96f7ebd358bda03ec42ea7c85a"
server_key_iv: "98aa85a0430e6757e83f02dacd5aeb84"
server_key_version: "encryptionkey@0"
I20260809 00:56:33.002450 2682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:33.031105 2682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:33.036465 2682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:33.036821 2682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:33.059907 2682 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:39741
I20260809 00:56:33.059880 2786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:39741 every 8 connection(s)
I20260809 00:56:33.061655 2682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:56:33.064062 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 2682
I20260809 00:56:33.064378 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:56:33.065176 10116 external_mini_cluster.cc:1428] Setting key e24646bcddc1f972978a14ee048de270
I20260809 00:56:33.067121 2787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:33.083817 2787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Bootstrap starting.
Aug 09 00:56:33 dist-test-slave-j2vl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:33.088743 2787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:33.090399 2787 log.cc:824] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:33.094671 2787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: No bootstrap required, opened a new log
I20260809 00:56:33.096024 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:33.067643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.9.225.62:39741)
0809 00:56:33.068657 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:33.068678 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:33.068743 (+ 65us) server_negotiation.cc:407] Connection header received
0809 00:56:33.071127 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:33.071166 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:33.071866 (+ 700us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:33.072450 (+ 584us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:33.075433 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:33.077693 (+ 2260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:33.079314 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:33.080555 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:33.080794 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:33.085665 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:33.085683 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:33.085710 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:33.085793 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:33.090318 (+ 4525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:33.091590 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:33.091599 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:33.091607 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:33.091809 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:33.092283 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:33.092288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:33.092291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:33.093059 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:33.093430 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:33.093932 (+ 502us) server_negotiation.cc:299] Negotiation successful
0809 00:56:33.094481 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":406,"threads_started":1}
I20260809 00:56:33.104223 2787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } }
I20260809 00:56:33.104722 2787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:33.104892 2787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6cac20e9c347b3a5c86bf76de4e24f, State: Initialized, Role: FOLLOWER
I20260809 00:56:33.105719 2787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } }
I20260809 00:56:33.105965 2787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:56:33.106074 2787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:56:33.106220 2787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:33.108858 2787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } }
I20260809 00:56:33.109628 2787 leader_election.cc:302] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c6cac20e9c347b3a5c86bf76de4e24f; no voters:
I20260809 00:56:33.110376 2787 leader_election.cc:288] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:56:33.110569 2793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:56:33.111276 2793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 1 LEADER]: Becoming Leader. State: Replica: 3c6cac20e9c347b3a5c86bf76de4e24f, State: Running, Role: LEADER
I20260809 00:56:33.112099 2793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } }
I20260809 00:56:33.112762 2787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:56:33.115034 2795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } } }
I20260809 00:56:33.115473 2795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:33.116219 2796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c6cac20e9c347b3a5c86bf76de4e24f. Latest consensus state: current_term: 1 leader_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } } }
I20260809 00:56:33.116617 2796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:33.118975 2801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:56:33.126122 2801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:56:33.136884 2801 catalog_manager.cc:1347] Generated new cluster ID: 9e1f2a5d7ba342cbb0d0b04512453327
I20260809 00:56:33.137072 2801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:56:33.148331 2801 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:56:33.149948 2801 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:56:33.163574 2801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Generated new TSK 0
I20260809 00:56:33.164660 2801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:56:33.262428 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:39741
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:56:33.670038 2821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:33.670440 2821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:33.670579 2821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:33.678592 2821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:33.678771 2821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:33.678898 2821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:56:33.694885 2821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:33.696444 2821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:33.698021 2821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:33.713737 2829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:33.713653 2830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:33.714145 2832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:33.714965 2821 server_base.cc:1034] running on GCE node
I20260809 00:56:33.715713 2821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:33.716910 2821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:33.718199 2821 hybrid_clock.cc:648] HybridClock initialized: now 1786236993718128 us; error 56 us; skew 500 ppm
I20260809 00:56:33.718573 2821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:33 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236993, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:33.722440 2821 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:56:33.724284 2821 webserver.cc:460] Webserver started at http://127.9.225.1:36477/ using document root <none> and password file <none>
I20260809 00:56:33.725142 2821 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:33.725311 2821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:33.725708 2821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:56:33.729136 2821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "28867c48c5374d46adab0381ff985ba3"
format_stamp: "Formatted at 2026-08-09 00:56:33 on dist-test-slave-j2vl"
server_key: "f1c73520ec68f49b4e6f3f84ada3346c"
server_key_iv: "1762f653fe98838b501a1e3e301b6725"
server_key_version: "encryptionkey@0"
I20260809 00:56:33.730140 2821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "28867c48c5374d46adab0381ff985ba3"
format_stamp: "Formatted at 2026-08-09 00:56:33 on dist-test-slave-j2vl"
server_key: "f1c73520ec68f49b4e6f3f84ada3346c"
server_key_iv: "1762f653fe98838b501a1e3e301b6725"
server_key_version: "encryptionkey@0"
I20260809 00:56:33.736593 2821 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 00:56:33.741027 2847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:33.742844 2821 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:56:33.743062 2821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "28867c48c5374d46adab0381ff985ba3"
format_stamp: "Formatted at 2026-08-09 00:56:33 on dist-test-slave-j2vl"
server_key: "f1c73520ec68f49b4e6f3f84ada3346c"
server_key_iv: "1762f653fe98838b501a1e3e301b6725"
server_key_version: "encryptionkey@0"
I20260809 00:56:33.743304 2821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:33.762861 2821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:33.769675 2821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:33.770035 2821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:33.771253 2821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:33.773011 2821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:33.773149 2821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:33.773310 2821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:33.773418 2821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260809 00:56:33.810647 2821 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:44589
I20260809 00:56:33.810691 3026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:44589 every 8 connection(s)
I20260809 00:56:33.812191 2821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:56:33.814389 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 2821
I20260809 00:56:33.814656 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:56:33.815495 10116 external_mini_cluster.cc:1428] Setting key dbed1f0ac642deb1644515ae87891e46
Aug 09 00:56:33 dist-test-slave-j2vl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236993, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:33.840364 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:33.815656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:60473 (local address 127.9.225.62:39741)
0809 00:56:33.816238 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:33.816245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:33.817486 (+ 1241us) server_negotiation.cc:407] Connection header received
0809 00:56:33.820587 (+ 3101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:33.820593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:33.820677 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:33.820808 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:33.823441 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:33.824265 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:33.827197 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:33.827637 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:33.827753 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:33.833078 (+ 5325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:33.833085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:33.833088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:33.833134 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:33.836362 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:33.837390 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:33.837397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:33.837408 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:33.837555 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:33.838102 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:33.838106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:33.838109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:33.838417 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:33.838592 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:33.839644 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0809 00:56:33.839958 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
I20260809 00:56:33.841661 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:33.816359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:60473)
0809 00:56:33.817258 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:33.817300 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:33.820158 (+ 2858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:33.821167 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:33.821183 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:33.822150 (+ 967us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:33.823141 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:33.823177 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:33.824492 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:33.824502 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:33.826959 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:33.826985 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:33.827938 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:33.827948 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:33.828340 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:33.829231 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:33.829273 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:33.832690 (+ 3417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:33.836600 (+ 3910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:33.836615 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:33.836643 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:33.837186 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:33.837692 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:33.837697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:33.837701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:33.837950 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:33.838567 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:33.838577 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:33.839145 (+ 568us) client_negotiation.cc:769] Sending connection context
0809 00:56:33.839503 (+ 358us) client_negotiation.cc:240] Negotiation successful
0809 00:56:33.840227 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":376,"threads_started":1}
I20260809 00:56:33.843858 3028 heartbeater.cc:347] Connected to a master server at 127.9.225.62:39741
I20260809 00:56:33.844331 3028 heartbeater.cc:464] Registering TS with master...
I20260809 00:56:33.845498 3028 heartbeater.cc:511] Master 127.9.225.62:39741 requested a full tablet report, sending...
I20260809 00:56:33.848732 2732 ts_manager.cc:194] Registered new tserver with Master: 28867c48c5374d46adab0381ff985ba3 (127.9.225.1:44589)
I20260809 00:56:33.851457 2732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:60473
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:56:33.898181 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:39741
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:56:34.286825 3034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:34.287220 3034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:34.287410 3034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:34.295195 3034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:34.295421 3034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:34.295622 3034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:56:34.306479 3034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:34.308256 3034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:34.309799 3034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:34.324965 3040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:34.324894 3043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:34.324928 3039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:34.325953 3034 server_base.cc:1034] running on GCE node
I20260809 00:56:34.326722 3034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:34.327827 3034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:34.329075 3034 hybrid_clock.cc:648] HybridClock initialized: now 1786236994329028 us; error 30 us; skew 500 ppm
I20260809 00:56:34.329535 3034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:34 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236994, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:34.333074 3034 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:56:34.334903 3034 webserver.cc:460] Webserver started at http://127.9.225.2:43065/ using document root <none> and password file <none>
I20260809 00:56:34.335714 3034 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:34.335881 3034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:34.336236 3034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:56:34.339701 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "4371a34cdec043f49d7e0f01795a469f"
format_stamp: "Formatted at 2026-08-09 00:56:34 on dist-test-slave-j2vl"
server_key: "d588176a98ee3c5aab7d019ebcd1006a"
server_key_iv: "25979e22386b69366c4988a1b356fca4"
server_key_version: "encryptionkey@0"
I20260809 00:56:34.340622 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "4371a34cdec043f49d7e0f01795a469f"
format_stamp: "Formatted at 2026-08-09 00:56:34 on dist-test-slave-j2vl"
server_key: "d588176a98ee3c5aab7d019ebcd1006a"
server_key_iv: "25979e22386b69366c4988a1b356fca4"
server_key_version: "encryptionkey@0"
I20260809 00:56:34.346809 3034 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260809 00:56:34.351965 3051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:34.354008 3034 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 00:56:34.354259 3034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "4371a34cdec043f49d7e0f01795a469f"
format_stamp: "Formatted at 2026-08-09 00:56:34 on dist-test-slave-j2vl"
server_key: "d588176a98ee3c5aab7d019ebcd1006a"
server_key_iv: "25979e22386b69366c4988a1b356fca4"
server_key_version: "encryptionkey@0"
I20260809 00:56:34.354559 3034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:34.379606 3034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:34.384801 3034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:34.385180 3034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:34.386384 3034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:34.387953 3034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:34.388072 3034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:34.388249 3034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:34.388327 3034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:34.427331 3034 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:40169
I20260809 00:56:34.427439 3228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:40169 every 8 connection(s)
I20260809 00:56:34.428821 3034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:56:34.429461 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 3034
I20260809 00:56:34.429669 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:56:34.430393 10116 external_mini_cluster.cc:1428] Setting key ffa23d40b2c4167081572bb496fb2a40
Aug 09 00:56:34 dist-test-slave-j2vl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236994, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:34.455084 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:34.432186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:54499 (local address 127.9.225.62:39741)
0809 00:56:34.432882 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:34.432889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:34.434478 (+ 1589us) server_negotiation.cc:407] Connection header received
0809 00:56:34.436555 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:34.436564 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:34.436676 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:34.436812 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:34.438930 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:34.440138 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:34.442155 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:34.442951 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:34.443118 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:34.448054 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:34.448063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:34.448067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:34.448127 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:34.451258 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:34.452251 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:34.452256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:34.452258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:34.452411 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:34.452939 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:34.452944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:34.452946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:34.453266 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:34.453487 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:34.454340 (+ 853us) server_negotiation.cc:299] Negotiation successful
0809 00:56:34.454641 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":444,"threads_started":1}
I20260809 00:56:34.456275 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:34.432909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:54499)
0809 00:56:34.434257 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:34.434318 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:34.436223 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:34.437131 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:34.437141 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:34.438006 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:34.438689 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:34.438710 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:34.440448 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:34.440454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:34.441973 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:34.441988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:34.443247 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:34.443253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:34.443580 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:34.444472 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:34.444500 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:34.447560 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:34.451489 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:34.451502 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:34.451533 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:34.452080 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:34.452540 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:34.452546 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:34.452550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:34.452787 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:34.453452 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:34.453462 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:34.453915 (+ 453us) client_negotiation.cc:769] Sending connection context
0809 00:56:34.454230 (+ 315us) client_negotiation.cc:240] Negotiation successful
0809 00:56:34.454926 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"thread_start_us":715,"threads_started":1}
I20260809 00:56:34.458572 3230 heartbeater.cc:347] Connected to a master server at 127.9.225.62:39741
I20260809 00:56:34.459059 3230 heartbeater.cc:464] Registering TS with master...
I20260809 00:56:34.460114 3230 heartbeater.cc:511] Master 127.9.225.62:39741 requested a full tablet report, sending...
I20260809 00:56:34.462456 2732 ts_manager.cc:194] Registered new tserver with Master: 4371a34cdec043f49d7e0f01795a469f (127.9.225.2:40169)
I20260809 00:56:34.463837 2732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:54499
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:56:34.508951 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:39741
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:56:34.855672 3028 heartbeater.cc:503] Master 127.9.225.62:39741 was elected leader, sending a full tablet report...
W20260809 00:56:34.895602 3237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:34.895961 3237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:34.896143 3237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:34.904057 3237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:34.904199 3237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:34.904395 3237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:56:34.915329 3237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:34.916919 3237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:34.918555 3237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:34.934049 3245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:34.934095 3247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:34.934316 3244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:34.935053 3237 server_base.cc:1034] running on GCE node
I20260809 00:56:34.935792 3237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:34.936844 3237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:34.938126 3237 hybrid_clock.cc:648] HybridClock initialized: now 1786236994938064 us; error 53 us; skew 500 ppm
I20260809 00:56:34.938537 3237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:34 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236994, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:34.942288 3237 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:56:34.944159 3237 webserver.cc:460] Webserver started at http://127.9.225.3:35955/ using document root <none> and password file <none>
I20260809 00:56:34.944921 3237 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:34.945036 3237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:34.945423 3237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:56:34.949024 3237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "a82a8a05b3794f9da2209fdfe0733596"
format_stamp: "Formatted at 2026-08-09 00:56:34 on dist-test-slave-j2vl"
server_key: "f12b031a780fda11cc4df99bd9f59a38"
server_key_iv: "06c5e632d77f5dc509ac5cd238391d32"
server_key_version: "encryptionkey@0"
I20260809 00:56:34.950002 3237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "a82a8a05b3794f9da2209fdfe0733596"
format_stamp: "Formatted at 2026-08-09 00:56:34 on dist-test-slave-j2vl"
server_key: "f12b031a780fda11cc4df99bd9f59a38"
server_key_iv: "06c5e632d77f5dc509ac5cd238391d32"
server_key_version: "encryptionkey@0"
I20260809 00:56:34.956676 3237 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260809 00:56:34.961495 3257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:34.963148 3237 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260809 00:56:34.963367 3237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "a82a8a05b3794f9da2209fdfe0733596"
format_stamp: "Formatted at 2026-08-09 00:56:34 on dist-test-slave-j2vl"
server_key: "f12b031a780fda11cc4df99bd9f59a38"
server_key_iv: "06c5e632d77f5dc509ac5cd238391d32"
server_key_version: "encryptionkey@0"
I20260809 00:56:34.963630 3237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:34.983258 3237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:34.988179 3237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:34.988521 3237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:34.989722 3237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:34.991350 3237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:34.991434 3237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:34.991619 3237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:34.991684 3237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:35.030546 3237 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:46747
I20260809 00:56:35.030575 3421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:46747 every 8 connection(s)
I20260809 00:56:35.032214 3237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:56:35.035670 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 3237
I20260809 00:56:35.035887 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:56:35.036538 10116 external_mini_cluster.cc:1428] Setting key db0129305225f03be667d3b1f3dfb012
Aug 09 00:56:35 dist-test-slave-j2vl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236994, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:35.056407 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.035769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:43311 (local address 127.9.225.62:39741)
0809 00:56:35.036381 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:35.036388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:35.037670 (+ 1282us) server_negotiation.cc:407] Connection header received
0809 00:56:35.040129 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:35.040135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:35.040227 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:35.040365 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:35.042141 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.042977 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.044672 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.045099 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.045245 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.049311 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:35.049317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:35.049321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:35.049414 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:35.052912 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:35.054021 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:35.054027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:35.054029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:35.054181 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:35.054635 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:35.054639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:35.054641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:35.054961 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:35.055090 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:35.055895 (+ 805us) server_negotiation.cc:299] Negotiation successful
0809 00:56:35.056101 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":306,"threads_started":1}
I20260809 00:56:35.057600 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.036440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:43311)
0809 00:56:35.037435 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:35.037518 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:35.039770 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:35.040642 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:35.040652 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:35.041224 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:35.041960 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:35.041980 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.043170 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.043174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:35.044509 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:35.044525 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.045390 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.045402 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:35.045670 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.046497 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:35.046549 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:35.049045 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:35.053099 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:35.053113 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:35.053136 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:35.053872 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:35.054284 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:35.054287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:35.054290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:35.054520 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:35.055096 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:35.055104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:35.055510 (+ 406us) client_negotiation.cc:769] Sending connection context
0809 00:56:35.055785 (+ 275us) client_negotiation.cc:240] Negotiation successful
0809 00:56:35.056250 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":392,"threads_started":1}
I20260809 00:56:35.059662 3423 heartbeater.cc:347] Connected to a master server at 127.9.225.62:39741
I20260809 00:56:35.060384 3423 heartbeater.cc:464] Registering TS with master...
I20260809 00:56:35.061472 3423 heartbeater.cc:511] Master 127.9.225.62:39741 requested a full tablet report, sending...
I20260809 00:56:35.063367 2732 ts_manager.cc:194] Registered new tserver with Master: a82a8a05b3794f9da2209fdfe0733596 (127.9.225.3:46747)
I20260809 00:56:35.064543 2732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:43311
I20260809 00:56:35.066519 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:56:35.088624 3430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.072378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.9.225.62:39741)
0809 00:56:35.072663 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:35.072672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:35.073081 (+ 409us) server_negotiation.cc:407] Connection header received
0809 00:56:35.073197 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:35.073202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:35.073283 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:35.073479 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:35.075155 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.075990 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.077577 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.078085 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.078204 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.081770 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:35.081777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:35.081780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:35.081844 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:35.084798 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:35.086130 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:35.086136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:35.086138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:35.086297 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:35.086983 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:35.086987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:35.086990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:35.087375 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:35.087585 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:35.087986 (+ 401us) server_negotiation.cc:299] Negotiation successful
0809 00:56:35.088208 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 00:56:35.093678 2732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:56:35.101166 2732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:56:35.128531 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.115426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:40169 (local address 127.0.0.1:60792)
0809 00:56:35.116453 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:35.116520 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:35.116758 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:35.118953 (+ 2195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:35.118961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:35.119005 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:35.119589 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:35.119606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.121840 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.121847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:35.124383 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:35.124405 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.126544 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.126554 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:35.127581 (+ 1027us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.127683 (+ 102us) client_negotiation.cc:769] Sending connection context
0809 00:56:35.127857 (+ 174us) client_negotiation.cc:240] Negotiation successful
0809 00:56:35.128082 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":625,"threads_started":1}
I20260809 00:56:35.129827 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.115877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.9.225.2:40169)
0809 00:56:35.118213 (+ 2336us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:35.118222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:35.118253 (+ 31us) server_negotiation.cc:407] Connection header received
0809 00:56:35.118375 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:35.118385 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:35.118675 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:35.119827 (+ 1152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:35.120081 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.121651 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.124664 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.126273 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.126725 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.126992 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:35.129238 (+ 2246us) server_negotiation.cc:299] Negotiation successful
0809 00:56:35.129495 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2140,"thread_start_us":2012,"threads_started":1}
I20260809 00:56:35.132311 3457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.115792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:44589 (local address 127.0.0.1:36334)
0809 00:56:35.118539 (+ 2747us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:35.118567 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:35.118743 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:35.119668 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:35.119675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:35.119706 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:35.120594 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:35.120608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.124210 (+ 3602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.124217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:35.128814 (+ 4597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:35.128832 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.131375 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.131381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:35.131600 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.131634 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 00:56:35.131736 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 00:56:35.131906 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2561,"thread_start_us":2346,"threads_started":1}
I20260809 00:56:35.132882 3459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.116339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36334 (local address 127.9.225.1:44589)
0809 00:56:35.117694 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:35.117709 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:35.118768 (+ 1059us) server_negotiation.cc:407] Connection header received
0809 00:56:35.118907 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:35.118914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:35.119137 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:35.119403 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:35.122277 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.124037 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.129073 (+ 5036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.130773 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.130920 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.131207 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:35.131881 (+ 674us) server_negotiation.cc:299] Negotiation successful
0809 00:56:35.132152 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114,"thread_start_us":902,"threads_started":1}
I20260809 00:56:35.137682 2918 tablet_service.cc:1467] Processing CreateTablet for tablet 770af34b6ebe47ad85a5273f45bc603a (DEFAULT_TABLE table=test-table [id=a99fc935360b4c96b1a8801981204981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:56:35.137684 3132 tablet_service.cc:1467] Processing CreateTablet for tablet 770af34b6ebe47ad85a5273f45bc603a (DEFAULT_TABLE table=test-table [id=a99fc935360b4c96b1a8801981204981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:56:35.138450 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.117145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:46747 (local address 127.0.0.1:58132)
0809 00:56:35.128860 (+ 11715us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:35.128883 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:35.129010 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:35.130944 (+ 1934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:35.130949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:35.130983 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:35.131180 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:35.131193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.133631 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.133636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:35.135657 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:35.135684 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.137613 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.137619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:35.137815 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.137854 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 00:56:35.137956 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 00:56:35.138119 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11603,"thread_start_us":15639,"threads_started":1}
I20260809 00:56:35.138510 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.117744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58132 (local address 127.9.225.3:46747)
0809 00:56:35.127226 (+ 9482us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:35.127237 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:35.130099 (+ 2862us) server_negotiation.cc:407] Connection header received
0809 00:56:35.130309 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:35.130319 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:35.130670 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:35.131393 (+ 723us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:35.131731 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.133355 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.135984 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.137438 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.137599 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.137852 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:35.138004 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 00:56:35.138177 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9168,"thread_start_us":8999,"threads_started":1}
I20260809 00:56:35.139401 2918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770af34b6ebe47ad85a5273f45bc603a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:35.139438 3132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770af34b6ebe47ad85a5273f45bc603a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:35.142334 3334 tablet_service.cc:1467] Processing CreateTablet for tablet 770af34b6ebe47ad85a5273f45bc603a (DEFAULT_TABLE table=test-table [id=a99fc935360b4c96b1a8801981204981]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:56:35.144459 3334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 770af34b6ebe47ad85a5273f45bc603a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:35.157438 3469 tablet_bootstrap.cc:492] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596: Bootstrap starting.
I20260809 00:56:35.162289 3469 tablet_bootstrap.cc:654] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:35.162377 3464 tablet_bootstrap.cc:492] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f: Bootstrap starting.
I20260809 00:56:35.163530 3469 log.cc:824] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:35.166302 3465 tablet_bootstrap.cc:492] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3: Bootstrap starting.
I20260809 00:56:35.167282 3464 tablet_bootstrap.cc:654] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:35.167999 3469 tablet_bootstrap.cc:492] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596: No bootstrap required, opened a new log
I20260809 00:56:35.168450 3469 ts_tablet_manager.cc:1397] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260809 00:56:35.168732 3464 log.cc:824] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:35.171078 3465 tablet_bootstrap.cc:654] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:35.172749 3465 log.cc:824] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:35.176687 3464 tablet_bootstrap.cc:492] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f: No bootstrap required, opened a new log
I20260809 00:56:35.176687 3465 tablet_bootstrap.cc:492] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3: No bootstrap required, opened a new log
I20260809 00:56:35.177065 3465 ts_tablet_manager.cc:1397] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260809 00:56:35.177065 3464 ts_tablet_manager.cc:1397] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260809 00:56:35.178359 3469 raft_consensus.cc:348] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } } peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } }
I20260809 00:56:35.178797 3469 raft_consensus.cc:374] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:35.178900 3469 raft_consensus.cc:729] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82a8a05b3794f9da2209fdfe0733596, State: Initialized, Role: FOLLOWER
I20260809 00:56:35.179682 3469 consensus_queue.cc:260] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } } peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } }
I20260809 00:56:35.180948 3423 heartbeater.cc:503] Master 127.9.225.62:39741 was elected leader, sending a full tablet report...
I20260809 00:56:35.181823 3469 ts_tablet_manager.cc:1428] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260809 00:56:35.188905 3464 raft_consensus.cc:348] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } } peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } }
I20260809 00:56:35.188900 3465 raft_consensus.cc:348] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } } peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } }
I20260809 00:56:35.189519 3465 raft_consensus.cc:374] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:35.189518 3464 raft_consensus.cc:374] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:35.189639 3465 raft_consensus.cc:729] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28867c48c5374d46adab0381ff985ba3, State: Initialized, Role: FOLLOWER
I20260809 00:56:35.189639 3464 raft_consensus.cc:729] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4371a34cdec043f49d7e0f01795a469f, State: Initialized, Role: FOLLOWER
I20260809 00:56:35.190413 3465 consensus_queue.cc:260] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } } peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } }
I20260809 00:56:35.190402 3464 consensus_queue.cc:260] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } } peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } }
I20260809 00:56:35.191704 3230 heartbeater.cc:503] Master 127.9.225.62:39741 was elected leader, sending a full tablet report...
I20260809 00:56:35.192713 3464 ts_tablet_manager.cc:1428] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260809 00:56:35.192750 3465 ts_tablet_manager.cc:1428] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3: Time spent starting tablet: real 0.016s user 0.016s sys 0.001s
W20260809 00:56:35.284225 3427 tablet.cc:2369] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:56:35.315382 3029 tablet.cc:2369] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:56:35.432991 3231 tablet.cc:2369] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:56:35.465611 3479 raft_consensus.cc:482] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:56:35.466065 3479 raft_consensus.cc:504] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } } peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } }
I20260809 00:56:35.468168 3479 leader_election.cc:288] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28867c48c5374d46adab0381ff985ba3 (127.9.225.1:44589), a82a8a05b3794f9da2209fdfe0733596 (127.9.225.3:46747)
I20260809 00:56:35.478343 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.468827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:33331 (local address 127.9.225.3:46747)
0809 00:56:35.469087 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:35.469094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:35.470043 (+ 949us) server_negotiation.cc:407] Connection header received
0809 00:56:35.470293 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:35.470299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:35.470380 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:35.470477 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:35.471335 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.472223 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.475440 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.476826 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.476991 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.477116 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:35.477898 (+ 782us) server_negotiation.cc:299] Negotiation successful
0809 00:56:35.478062 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260809 00:56:35.478348 3459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.468911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:55247 (local address 127.9.225.1:44589)
0809 00:56:35.469125 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:35.469131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:35.469921 (+ 790us) server_negotiation.cc:407] Connection header received
0809 00:56:35.470236 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:35.470244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:35.470336 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:35.470465 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:35.471376 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.472116 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.475439 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.476826 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.476991 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.477116 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:35.477898 (+ 782us) server_negotiation.cc:299] Negotiation successful
0809 00:56:35.478063 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 00:56:35.478613 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.468639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:46747 (local address 127.9.225.2:33331)
0809 00:56:35.469937 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:35.469961 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:35.470147 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:35.470555 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:35.470561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:35.470596 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:35.471116 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:35.471132 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.472359 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.472363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:35.475172 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:35.475197 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.477213 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.477218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:35.477555 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.477672 (+ 117us) client_negotiation.cc:769] Sending connection context
0809 00:56:35.477792 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 00:56:35.478010 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"mutex_wait_us":342,"thread_start_us":866,"threads_started":1}
I20260809 00:56:35.478654 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.468639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:44589 (local address 127.9.225.2:55247)
0809 00:56:35.469752 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:35.469791 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:35.470048 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:35.470555 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:35.470561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:35.470668 (+ 107us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:35.471165 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:35.471176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.472272 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.472279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:35.475172 (+ 2893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:35.475197 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.477200 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.477208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:35.477581 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.477672 (+ 91us) client_negotiation.cc:769] Sending connection context
0809 00:56:35.477792 (+ 120us) client_negotiation.cc:240] Negotiation successful
0809 00:56:35.478010 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":280,"spinlock_wait_cycles":27392,"thread_start_us":669,"threads_started":1}
I20260809 00:56:35.480007 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770af34b6ebe47ad85a5273f45bc603a" candidate_uuid: "4371a34cdec043f49d7e0f01795a469f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82a8a05b3794f9da2209fdfe0733596" is_pre_election: true
I20260809 00:56:35.480007 2954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770af34b6ebe47ad85a5273f45bc603a" candidate_uuid: "4371a34cdec043f49d7e0f01795a469f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28867c48c5374d46adab0381ff985ba3" is_pre_election: true
I20260809 00:56:35.480657 3360 raft_consensus.cc:2454] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4371a34cdec043f49d7e0f01795a469f in term 0.
I20260809 00:56:35.480662 2954 raft_consensus.cc:2454] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4371a34cdec043f49d7e0f01795a469f in term 0.
I20260809 00:56:35.481582 3053 leader_election.cc:302] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28867c48c5374d46adab0381ff985ba3, 4371a34cdec043f49d7e0f01795a469f, a82a8a05b3794f9da2209fdfe0733596; no voters:
I20260809 00:56:35.482116 3479 raft_consensus.cc:2781] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:56:35.482322 3479 raft_consensus.cc:482] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:56:35.482453 3479 raft_consensus.cc:3037] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:35.484825 3479 raft_consensus.cc:504] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } } peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } }
I20260809 00:56:35.485675 3479 leader_election.cc:288] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [CANDIDATE]: Term 1 election: Requested vote from peers 28867c48c5374d46adab0381ff985ba3 (127.9.225.1:44589), a82a8a05b3794f9da2209fdfe0733596 (127.9.225.3:46747)
I20260809 00:56:35.486347 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770af34b6ebe47ad85a5273f45bc603a" candidate_uuid: "4371a34cdec043f49d7e0f01795a469f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82a8a05b3794f9da2209fdfe0733596"
I20260809 00:56:35.486377 2954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "770af34b6ebe47ad85a5273f45bc603a" candidate_uuid: "4371a34cdec043f49d7e0f01795a469f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28867c48c5374d46adab0381ff985ba3"
I20260809 00:56:35.486588 3360 raft_consensus.cc:3037] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:35.486620 2954 raft_consensus.cc:3037] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:35.489650 2954 raft_consensus.cc:2454] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4371a34cdec043f49d7e0f01795a469f in term 1.
I20260809 00:56:35.489663 3360 raft_consensus.cc:2454] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4371a34cdec043f49d7e0f01795a469f in term 1.
I20260809 00:56:35.490262 3056 leader_election.cc:302] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28867c48c5374d46adab0381ff985ba3, 4371a34cdec043f49d7e0f01795a469f, a82a8a05b3794f9da2209fdfe0733596; no voters:
I20260809 00:56:35.490685 3479 raft_consensus.cc:2781] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:56:35.491266 3479 raft_consensus.cc:686] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 1 LEADER]: Becoming Leader. State: Replica: 4371a34cdec043f49d7e0f01795a469f, State: Running, Role: LEADER
I20260809 00:56:35.491828 3479 consensus_queue.cc:237] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } } peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } }
I20260809 00:56:35.497184 2732 catalog_manager.cc:5495] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4371a34cdec043f49d7e0f01795a469f (127.9.225.2). New cstate: current_term: 1 leader_uuid: "4371a34cdec043f49d7e0f01795a469f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } health_report { overall_health: UNKNOWN } } }
I20260809 00:56:35.591584 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.585579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.9.225.2:40169)
0809 00:56:35.585777 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:35.585782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:35.585950 (+ 168us) server_negotiation.cc:407] Connection header received
0809 00:56:35.586154 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:35.586158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:35.586248 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:35.586467 (+ 219us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:56:35.586895 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.587655 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.589221 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.589710 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.589814 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.590389 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:56:35.590992 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:56:35.591111 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:35.591244 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 00:56:35.591329 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 00:56:35.601403 3360 raft_consensus.cc:1261] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596 [term 1 FOLLOWER]: Refusing update from remote peer 4371a34cdec043f49d7e0f01795a469f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:56:35.601403 2954 raft_consensus.cc:1261] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3 [term 1 FOLLOWER]: Refusing update from remote peer 4371a34cdec043f49d7e0f01795a469f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:56:35.602705 3479 consensus_queue.cc:1035] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:56:35.603097 3483 consensus_queue.cc:1035] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:56:35.617079 3491 mvcc.cc:205] Tried to move back new op lower bound from 7316426733971197952 to 7316426733539405824. Current Snapshot: MvccSnapshot[applied={T|T < 7316426733971197952}]
I20260809 00:56:35.618486 3489 mvcc.cc:205] Tried to move back new op lower bound from 7316426733971197952 to 7316426733539405824. Current Snapshot: MvccSnapshot[applied={T|T < 7316426733971197952}]
I20260809 00:56:35.622373 3492 mvcc.cc:205] Tried to move back new op lower bound from 7316426733971197952 to 7316426733539405824. Current Snapshot: MvccSnapshot[applied={T|T < 7316426733971197952}]
I20260809 00:56:35.635056 2732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:56:35.635371 2732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:56:35.639389 2732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Sending DeleteTablet for 3 replicas of tablet 770af34b6ebe47ad85a5273f45bc603a
I20260809 00:56:35.640851 2918 tablet_service.cc:1514] Processing DeleteTablet for tablet 770af34b6ebe47ad85a5273f45bc603a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:56:35 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:36334
I20260809 00:56:35.640865 3132 tablet_service.cc:1514] Processing DeleteTablet for tablet 770af34b6ebe47ad85a5273f45bc603a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:56:35 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:60792
I20260809 00:56:35.641093 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 2821
I20260809 00:56:35.642037 3334 tablet_service.cc:1514] Processing DeleteTablet for tablet 770af34b6ebe47ad85a5273f45bc603a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:56:35 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:58132
I20260809 00:56:35.658046 3517 tablet_replica.cc:330] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596: stopping tablet replica
I20260809 00:56:35.658066 3514 tablet_replica.cc:330] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3: stopping tablet replica
I20260809 00:56:35.658819 3517 raft_consensus.cc:2229] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:56:35.659687 3517 raft_consensus.cc:2258] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:56:35.660043 3514 raft_consensus.cc:2229] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:56:35.660965 3514 raft_consensus.cc:2258] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:56:35.662665 3517 ts_tablet_manager.cc:1905] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:56:35.664562 3514 ts_tablet_manager.cc:1905] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:56:35.665938 3516 tablet_replica.cc:330] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f: stopping tablet replica
I20260809 00:56:35.666661 3516 raft_consensus.cc:2229] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:56:35.667630 3516 raft_consensus.cc:2258] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:56:35.670696 3517 ts_tablet_manager.cc:1918] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:56:35.671003 3517 log.cc:1196] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/770af34b6ebe47ad85a5273f45bc603a
I20260809 00:56:35.671717 3517 ts_tablet_manager.cc:1939] T 770af34b6ebe47ad85a5273f45bc603a P a82a8a05b3794f9da2209fdfe0733596: Deleting consensus metadata
Aug 09 00:56:35 dist-test-slave-j2vl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:56:35.674094 2708 catalog_manager.cc:4841] TS a82a8a05b3794f9da2209fdfe0733596 (127.9.225.3:46747): tablet 770af34b6ebe47ad85a5273f45bc603a (table test-table [id=a99fc935360b4c96b1a8801981204981]) successfully deleted
I20260809 00:56:35.675225 3516 ts_tablet_manager.cc:1905] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:56:35.675433 3514 ts_tablet_manager.cc:1918] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:56:35.675689 3514 log.cc:1196] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/770af34b6ebe47ad85a5273f45bc603a
I20260809 00:56:35.676344 3514 ts_tablet_manager.cc:1939] T 770af34b6ebe47ad85a5273f45bc603a P 28867c48c5374d46adab0381ff985ba3: Deleting consensus metadata
I20260809 00:56:35.678615 2712 catalog_manager.cc:4841] TS 28867c48c5374d46adab0381ff985ba3 (127.9.225.1:44589): tablet 770af34b6ebe47ad85a5273f45bc603a (table test-table [id=a99fc935360b4c96b1a8801981204981]) successfully deleted
I20260809 00:56:35.682240 3459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.643922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.9.225.1:44589)
0809 00:56:35.644115 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:35.644121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:35.650718 (+ 6597us) server_negotiation.cc:407] Connection header received
0809 00:56:35.652217 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:35.652226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:35.652375 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:35.652468 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:35.657925 (+ 5457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.659156 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.661684 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.662375 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.662504 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.673746 (+ 11242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:35.673771 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:35.673792 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:35.673862 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:35.678277 (+ 4415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:35.679416 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:35.679429 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:35.679433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:35.679624 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:35.680156 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:35.680163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:35.680166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:35.680695 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:35.681055 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:35.681454 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 00:56:35.681815 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 00:56:35.683075 3018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:35.684566 3516 ts_tablet_manager.cc:1918] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:56:35.684890 3516 log.cc:1196] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/770af34b6ebe47ad85a5273f45bc603a
I20260809 00:56:35.685746 3516 ts_tablet_manager.cc:1939] T 770af34b6ebe47ad85a5273f45bc603a P 4371a34cdec043f49d7e0f01795a469f: Deleting consensus metadata
I20260809 00:56:35.687660 2708 catalog_manager.cc:4841] TS 4371a34cdec043f49d7e0f01795a469f (127.9.225.2:40169): tablet 770af34b6ebe47ad85a5273f45bc603a (table test-table [id=a99fc935360b4c96b1a8801981204981]) successfully deleted
I20260809 00:56:35.837615 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 2821
I20260809 00:56:35.867069 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 3034
Aug 09 00:56:35 dist-test-slave-j2vl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:56:35.887635 3458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:35.867989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.9.225.2:40169)
0809 00:56:35.868223 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:35.868230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:35.868256 (+ 26us) server_negotiation.cc:407] Connection header received
0809 00:56:35.868516 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:35.868522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:35.868622 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:35.868751 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:35.870777 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.872068 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.874072 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:35.874822 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:35.874974 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:35.879728 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:35.879750 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:35.879774 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:35.879848 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:35.884257 (+ 4409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:35.885159 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:35.885170 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:35.885175 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:35.885417 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:35.885839 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:35.885844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:35.885848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:35.886359 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:35.886749 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:35.886952 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 00:56:35.887213 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260809 00:56:35.888517 3223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:36.012223 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 3034
I20260809 00:56:36.041482 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 3237
Aug 09 00:56:36 dist-test-slave-j2vl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786236992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:56:36.060364 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:36.042424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.9.225.3:46747)
0809 00:56:36.043327 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:36.043332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:36.043350 (+ 18us) server_negotiation.cc:407] Connection header received
0809 00:56:36.043403 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:36.043407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:36.043482 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:36.043619 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:36.045101 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:36.046126 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:36.047564 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:36.048024 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:36.048175 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:36.053191 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:36.053210 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:36.053230 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:36.053276 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:36.056881 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:36.057918 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:36.057927 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:36.057930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:36.058093 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:36.058529 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:36.058535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:36.058540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:36.058974 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:36.059355 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:36.059675 (+ 320us) server_negotiation.cc:299] Negotiation successful
0809 00:56:36.059961 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":658,"threads_started":1}
I20260809 00:56:36.061120 3416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:36.154657 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 3237
I20260809 00:56:36.179809 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 2682
I20260809 00:56:36.181270 2780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:36.297812 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 2682
I20260809 00:56:56.333588 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39741
--webserver_interface=127.9.225.62
--webserver_port=43759
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:56:56.732771 3524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:56.733247 3524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:56.733469 3524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:56.741874 3524 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 00:56:56.742079 3524 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 00:56:56.742187 3524 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 00:56:56.742285 3524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:56:56.742393 3524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:56.742522 3524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:56.742617 3524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:56:56.742704 3524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:56:56.755263 3524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=43759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:56.756865 3524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:56.758428 3524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:56.789058 3533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:56.789122 3532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:56.789147 3535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:56.790274 3524 server_base.cc:1034] running on GCE node
I20260809 00:56:56.791648 3524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:56.794106 3524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:56.795477 3524 hybrid_clock.cc:648] HybridClock initialized: now 1786237016795385 us; error 59 us; skew 500 ppm
I20260809 00:56:56.796052 3524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:56 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237016, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:56.800731 3524 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:56:56.802995 3524 webserver.cc:460] Webserver started at http://127.9.225.62:43759/ using document root <none> and password file <none>
I20260809 00:56:56.803943 3524 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:56.804080 3524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:56.811779 3524 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.005s
I20260809 00:56:56.816094 3544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:56.818235 3524 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260809 00:56:56.818511 3524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "3c6cac20e9c347b3a5c86bf76de4e24f"
format_stamp: "Formatted at 2026-08-09 00:56:32 on dist-test-slave-j2vl"
server_key: "c86c6c96f7ebd358bda03ec42ea7c85a"
server_key_iv: "98aa85a0430e6757e83f02dacd5aeb84"
server_key_version: "encryptionkey@0"
I20260809 00:56:56.819744 3524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:56.839003 3524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:56.844661 3524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:56.845073 3524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:56.868016 3524 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:39741
I20260809 00:56:56.868042 3628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:39741 every 8 connection(s)
I20260809 00:56:56.870211 3524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:56:56.878844 3629 sys_catalog.cc:263] Verifying existing consensus state
I20260809 00:56:56.880679 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 3524
I20260809 00:56:56.881743 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:44589
--local_ip_for_outbound_sockets=127.9.225.1
--tserver_master_addrs=127.9.225.62:39741
--webserver_port=36477
--webserver_interface=127.9.225.1
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:56:56.883162 3629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Bootstrap starting.
I20260809 00:56:56.906668 3629 log.cc:824] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:56.920821 3629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 00:56:56.921579 3629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Bootstrap complete.
I20260809 00:56:56.933126 3629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } }
I20260809 00:56:56.934296 3629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6cac20e9c347b3a5c86bf76de4e24f, State: Initialized, Role: FOLLOWER
I20260809 00:56:56.935148 3629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } }
I20260809 00:56:56.935469 3629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:56:56.935626 3629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:56:56.935827 3629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 1 FOLLOWER]: Advancing to term 2
I20260809 00:56:56.940506 3629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } }
I20260809 00:56:56.941198 3629 leader_election.cc:302] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c6cac20e9c347b3a5c86bf76de4e24f; no voters:
I20260809 00:56:56.942095 3629 leader_election.cc:288] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 00:56:56.942272 3637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 00:56:56.943674 3637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 2 LEADER]: Becoming Leader. State: Replica: 3c6cac20e9c347b3a5c86bf76de4e24f, State: Running, Role: LEADER
I20260809 00:56:56.944487 3637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } }
I20260809 00:56:56.944918 3629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:56:56.947635 3639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c6cac20e9c347b3a5c86bf76de4e24f. Latest consensus state: current_term: 2 leader_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } } }
I20260809 00:56:56.948194 3639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:56.947670 3638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } } }
I20260809 00:56:56.948999 3638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: This master's current role is: LEADER
I20260809 00:56:56.957767 3645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:56:56.970706 3645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:56:56.973285 3645 catalog_manager.cc:1259] Loaded cluster ID: 9e1f2a5d7ba342cbb0d0b04512453327
I20260809 00:56:56.973584 3645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:56:56.979768 3645 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:56:56.982568 3645 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Loaded TSK: 0
I20260809 00:56:57.045055 3664 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 00:56:57.047771 3645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Generated new TSK 1
I20260809 00:56:57.048511 3645 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 00:56:57.357656 3635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:57.358081 3635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:57.358265 3635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:57.366255 3635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:57.366448 3635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:57.366633 3635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:56:57.378319 3635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:44589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=36477
--tserver_master_addrs=127.9.225.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:57.379954 3635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:57.381582 3635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:57.399405 3672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:57.399531 3674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:57.399796 3676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:57.400530 3635 server_base.cc:1034] running on GCE node
I20260809 00:56:57.401451 3635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:57.402864 3635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:57.404201 3635 hybrid_clock.cc:648] HybridClock initialized: now 1786237017404128 us; error 70 us; skew 500 ppm
I20260809 00:56:57.404632 3635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:57 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237017, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:57.409535 3635 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:56:57.411867 3635 webserver.cc:460] Webserver started at http://127.9.225.1:36477/ using document root <none> and password file <none>
I20260809 00:56:57.412958 3635 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:57.413142 3635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:57.421790 3635 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.004s
I20260809 00:56:57.426091 3684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:57.428054 3635 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:56:57.428323 3635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "28867c48c5374d46adab0381ff985ba3"
format_stamp: "Formatted at 2026-08-09 00:56:33 on dist-test-slave-j2vl"
server_key: "f1c73520ec68f49b4e6f3f84ada3346c"
server_key_iv: "1762f653fe98838b501a1e3e301b6725"
server_key_version: "encryptionkey@0"
I20260809 00:56:57.429705 3635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:57.460345 3635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:57.465790 3635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:57.466125 3635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:57.467252 3635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:57.468974 3635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:57.469131 3635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:57.469282 3635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:57.469390 3635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:57.504724 3635 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:44589
I20260809 00:56:57.504788 3850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:44589 every 8 connection(s)
I20260809 00:56:57.506438 3635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:56:57.513406 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 3635
I20260809 00:56:57.514128 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:40169
--local_ip_for_outbound_sockets=127.9.225.2
--tserver_master_addrs=127.9.225.62:39741
--webserver_port=43065
--webserver_interface=127.9.225.2
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:56:57 dist-test-slave-j2vl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237017, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:57.539309 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:57.509989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:41653)
0809 00:56:57.510836 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:57.510884 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:57.512823 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:57.514916 (+ 2093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:57.514931 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:57.515825 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:57.516722 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:57.516755 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:57.518702 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:57.518711 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:57.521036 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:57.521060 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:57.522355 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:57.522366 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:57.522693 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:57.523528 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:57.523567 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:57.527289 (+ 3722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:57.533730 (+ 6441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:57.533740 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:57.533757 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:57.534491 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:57.535219 (+ 728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:57.535247 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:57.535252 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:57.535506 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:57.536743 (+ 1237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:57.536751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:57.537147 (+ 396us) client_negotiation.cc:769] Sending connection context
0809 00:56:57.537508 (+ 361us) client_negotiation.cc:240] Negotiation successful
0809 00:56:57.538013 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":367,"threads_started":1}
I20260809 00:56:57.539624 3854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:57.509790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:41653 (local address 127.9.225.62:39741)
0809 00:56:57.510398 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:57.510411 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:57.511061 (+ 650us) server_negotiation.cc:407] Connection header received
0809 00:56:57.513781 (+ 2720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:57.513795 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:57.514184 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:57.514655 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:57.517019 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:57.518474 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:57.521255 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:57.522062 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:57.522231 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:57.527685 (+ 5454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:57.527702 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:57.527732 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:57.527812 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:57.533253 (+ 5441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:57.534734 (+ 1481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:57.534749 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:57.534760 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:57.535055 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:57.535641 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:57.535649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:57.535655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:57.536547 (+ 892us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:57.536804 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:57.537772 (+ 968us) server_negotiation.cc:299] Negotiation successful
0809 00:56:57.538305 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":283,"threads_started":1}
I20260809 00:56:57.542970 3851 heartbeater.cc:347] Connected to a master server at 127.9.225.62:39741
I20260809 00:56:57.543365 3851 heartbeater.cc:464] Registering TS with master...
I20260809 00:56:57.544344 3851 heartbeater.cc:511] Master 127.9.225.62:39741 requested a full tablet report, sending...
I20260809 00:56:57.547354 3570 ts_manager.cc:194] Registered new tserver with Master: 28867c48c5374d46adab0381ff985ba3 (127.9.225.1:44589)
I20260809 00:56:57.549099 3570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:41653
W20260809 00:56:57.921785 3856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:57.922223 3856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:57.922420 3856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:57.930871 3856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:57.931084 3856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:57.931273 3856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:56:57.942608 3856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:40169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=43065
--tserver_master_addrs=127.9.225.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:57.944209 3856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:57.945807 3856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:57.961467 3867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:57.961417 3865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:57.961508 3864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:57.962498 3856 server_base.cc:1034] running on GCE node
I20260809 00:56:57.963279 3856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:57.964401 3856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:57.965672 3856 hybrid_clock.cc:648] HybridClock initialized: now 1786237017965602 us; error 73 us; skew 500 ppm
I20260809 00:56:57.966120 3856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:57 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237017, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:57.969835 3856 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:56:57.971624 3856 webserver.cc:460] Webserver started at http://127.9.225.2:43065/ using document root <none> and password file <none>
I20260809 00:56:57.972404 3856 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:57.972536 3856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:57.978826 3856 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 00:56:57.982577 3880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:57.984457 3856 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 00:56:57.984701 3856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "4371a34cdec043f49d7e0f01795a469f"
format_stamp: "Formatted at 2026-08-09 00:56:34 on dist-test-slave-j2vl"
server_key: "d588176a98ee3c5aab7d019ebcd1006a"
server_key_iv: "25979e22386b69366c4988a1b356fca4"
server_key_version: "encryptionkey@0"
I20260809 00:56:57.985937 3856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:58.003159 3856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:58.008328 3856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:58.008697 3856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:58.009984 3856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:58.011647 3856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:58.011746 3856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:58.011933 3856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:58.012044 3856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:58.050025 3856 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:40169
I20260809 00:56:58.050086 4051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:40169 every 8 connection(s)
I20260809 00:56:58.051672 3856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:56:58.056957 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 3856
I20260809 00:56:58.057717 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:46747
--local_ip_for_outbound_sockets=127.9.225.3
--tserver_master_addrs=127.9.225.62:39741
--webserver_port=35955
--webserver_interface=127.9.225.3
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:56:58 dist-test-slave-j2vl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237017, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:58.076475 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.054808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:60069 (local address 127.9.225.62:39741)
0809 00:56:58.055345 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:58.055350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:58.056424 (+ 1074us) server_negotiation.cc:407] Connection header received
0809 00:56:58.058660 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:58.058666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:58.058775 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:58.058911 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:58.060815 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.061666 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.063444 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.063875 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.064026 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.068396 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:58.068403 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:58.068407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:58.068477 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:58.072663 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:58.073733 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:58.073746 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:58.073749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:58.073910 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:58.074451 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:58.074455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:58.074458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:58.074778 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:58.074956 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:58.075775 (+ 819us) server_negotiation.cc:299] Negotiation successful
0809 00:56:58.076076 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":257,"threads_started":1}
I20260809 00:56:58.077709 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.055412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:60069)
0809 00:56:58.056200 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:58.056243 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:58.058317 (+ 2074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:58.059213 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:58.059223 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:58.059857 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:58.060608 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.060632 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.061871 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.061875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.063241 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.063255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.064163 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.064172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.064441 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.065259 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:58.065285 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:58.068063 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:58.072893 (+ 4830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:58.072906 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:58.072937 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:58.073577 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:58.074044 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:58.074050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:58.074054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:58.074292 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:58.074927 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:58.074937 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:58.075368 (+ 431us) client_negotiation.cc:769] Sending connection context
0809 00:56:58.075665 (+ 297us) client_negotiation.cc:240] Negotiation successful
0809 00:56:58.076307 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":293,"threads_started":1}
I20260809 00:56:58.079551 4057 heartbeater.cc:347] Connected to a master server at 127.9.225.62:39741
I20260809 00:56:58.079934 4057 heartbeater.cc:464] Registering TS with master...
I20260809 00:56:58.080927 4057 heartbeater.cc:511] Master 127.9.225.62:39741 requested a full tablet report, sending...
I20260809 00:56:58.082921 3570 ts_manager.cc:194] Registered new tserver with Master: 4371a34cdec043f49d7e0f01795a469f (127.9.225.2:40169)
I20260809 00:56:58.084358 3570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:60069
W20260809 00:56:58.442010 4063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:56:58.442355 4063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:56:58.442530 4063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:56:58.450711 4063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:56:58.450870 4063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:56:58.451071 4063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:56:58.462080 4063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:46747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=35955
--tserver_master_addrs=127.9.225.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:56:58.463719 4063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:56:58.465293 4063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:56:58.483866 4074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:58.484035 4073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:56:58.483942 4076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:56:58.485286 4063 server_base.cc:1034] running on GCE node
I20260809 00:56:58.486092 4063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:56:58.487198 4063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:56:58.490622 4063 hybrid_clock.cc:648] HybridClock initialized: now 1786237018490515 us; error 176 us; skew 500 ppm
I20260809 00:56:58.491153 4063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:56:58 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237018, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:56:58.495131 4063 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:56:58.497014 4063 webserver.cc:460] Webserver started at http://127.9.225.3:35955/ using document root <none> and password file <none>
I20260809 00:56:58.497932 4063 fs_manager.cc:359] Metadata directory not provided
I20260809 00:56:58.498109 4063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:56:58.504843 4063 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 00:56:58.508378 4086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:58.510238 4063 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:56:58.510427 4063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "a82a8a05b3794f9da2209fdfe0733596"
format_stamp: "Formatted at 2026-08-09 00:56:34 on dist-test-slave-j2vl"
server_key: "f12b031a780fda11cc4df99bd9f59a38"
server_key_iv: "06c5e632d77f5dc509ac5cd238391d32"
server_key_version: "encryptionkey@0"
I20260809 00:56:58.511512 4063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:56:58.534909 4063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:56:58.539691 4063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:56:58.539964 4063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:56:58.541525 4063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:56:58.543136 4063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:56:58.543251 4063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:58.543339 4063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:56:58.543397 4063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:56:58.554013 3851 heartbeater.cc:503] Master 127.9.225.62:39741 was elected leader, sending a full tablet report...
I20260809 00:56:58.586113 4063 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:46747
I20260809 00:56:58.586171 4258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:46747 every 8 connection(s)
I20260809 00:56:58.587823 4063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:56:58.597297 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 4063
Aug 09 00:56:58 dist-test-slave-j2vl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237018, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:56:58.615091 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.591065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:53825 (local address 127.9.225.62:39741)
0809 00:56:58.591622 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:58.591627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:58.592693 (+ 1066us) server_negotiation.cc:407] Connection header received
0809 00:56:58.595660 (+ 2967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:58.595668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:58.595810 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:58.595955 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:58.598192 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.599473 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.601700 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.602444 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.602569 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.607715 (+ 5146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:58.607721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:58.607724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:58.607760 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:58.610942 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:58.612142 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:58.612147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:58.612149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:58.612300 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:58.612940 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:58.612944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:58.612947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:58.613263 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:58.613529 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:58.614373 (+ 844us) server_negotiation.cc:299] Negotiation successful
0809 00:56:58.614632 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":244,"threads_started":1}
I20260809 00:56:58.616020 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.598683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44012 (local address 127.9.225.62:39741)
0809 00:56:58.599747 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:58.599752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:58.599773 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:56:58.599829 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:58.599836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:58.599936 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:58.600067 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:58.602339 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.603579 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.605143 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.605841 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.605995 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.609445 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:58.609452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:58.609456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:58.609508 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:58.612904 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:58.613807 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:58.613813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:58.613817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:58.613963 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:58.614384 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:58.614392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:58.614395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:58.614671 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:58.615068 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:58.615291 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 00:56:58.615535 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":920,"thread_start_us":729,"threads_started":1}
I20260809 00:56:58.617012 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.591689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:53825)
0809 00:56:58.592488 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:58.592540 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:58.595240 (+ 2700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:58.596305 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:58.596315 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:58.597037 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:56:58.597934 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.597959 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.599748 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.599755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.601508 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.601528 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.602761 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.602773 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.603157 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.604129 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:56:58.604171 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:56:58.607412 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:56:58.611149 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:58.611164 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:58.611191 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:58.611988 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:58.612445 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:56:58.612450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:56:58.612454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:56:58.612756 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:56:58.613496 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:56:58.613506 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:56:58.613955 (+ 449us) client_negotiation.cc:769] Sending connection context
0809 00:56:58.614285 (+ 330us) client_negotiation.cc:240] Negotiation successful
0809 00:56:58.614996 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":296,"threads_started":1}
I20260809 00:56:58.618712 4259 heartbeater.cc:347] Connected to a master server at 127.9.225.62:39741
I20260809 00:56:58.619107 4259 heartbeater.cc:464] Registering TS with master...
I20260809 00:56:58.619959 4259 heartbeater.cc:511] Master 127.9.225.62:39741 requested a full tablet report, sending...
I20260809 00:56:58.621778 3568 ts_manager.cc:194] Registered new tserver with Master: a82a8a05b3794f9da2209fdfe0733596 (127.9.225.3:46747)
I20260809 00:56:58.622797 3568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:53825
I20260809 00:56:58.634238 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:56:58.642104 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.635623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.9.225.62:39741)
0809 00:56:58.635808 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:58.635813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:58.636207 (+ 394us) server_negotiation.cc:407] Connection header received
0809 00:56:58.636421 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:58.636425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:58.636604 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:58.636758 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:56:58.637254 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.638125 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.639757 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.640218 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.640333 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.641011 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:56:58.641328 (+ 317us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 00:56:58.641871 (+ 543us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44026: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 00:56:58.642277 4266 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44026: token expired
W20260809 00:56:58.643128 3434 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.225.62:39741)
I20260809 00:56:58.658686 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.643834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.9.225.62:39741)
0809 00:56:58.644132 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:58.644136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:58.644175 (+ 39us) server_negotiation.cc:407] Connection header received
0809 00:56:58.644225 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:58.644229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:58.644291 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:58.644373 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:58.646239 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.647220 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.648708 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.649150 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.649268 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.652453 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:58.652459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:58.652463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:58.652508 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:58.655449 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:58.656342 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:58.656346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:58.656349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:58.656520 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:58.657038 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:58.657041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:58.657043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:58.657536 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:58.657748 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:58.658208 (+ 460us) server_negotiation.cc:299] Negotiation successful
0809 00:56:58.658444 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260809 00:56:58.662626 3568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:56:58.670828 3568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:56:58.708441 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.685900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.9.225.1:44589)
0809 00:56:58.697114 (+ 11214us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:58.697129 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:58.698158 (+ 1029us) server_negotiation.cc:407] Connection header received
0809 00:56:58.698283 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:58.698294 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:58.698618 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:58.699801 (+ 1183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:58.700098 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.702377 (+ 2279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.705316 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.707041 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.707190 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.707471 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:58.707886 (+ 415us) server_negotiation.cc:299] Negotiation successful
0809 00:56:58.708087 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10995,"thread_start_us":10815,"threads_started":1}
I20260809 00:56:58.708514 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.685589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:44589 (local address 127.0.0.1:59474)
0809 00:56:58.697735 (+ 12146us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:58.697784 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:58.698028 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:58.698973 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:58.698982 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:58.699022 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:58.699567 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.699581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.702524 (+ 2943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.702529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.705146 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.705160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.707326 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.707334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.707569 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.707637 (+ 68us) client_negotiation.cc:769] Sending connection context
0809 00:56:58.707804 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 00:56:58.708033 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11855,"mutex_wait_us":68,"thread_start_us":11637,"threads_started":1}
I20260809 00:56:58.709131 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.686257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.9.225.3:46747)
0809 00:56:58.698815 (+ 12558us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:58.698826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:58.699295 (+ 469us) server_negotiation.cc:407] Connection header received
0809 00:56:58.699435 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:58.699444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:58.699841 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:58.700074 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:58.701133 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.702747 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.706171 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.707210 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.707357 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.707595 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:58.708162 (+ 567us) server_negotiation.cc:299] Negotiation successful
0809 00:56:58.708347 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12337,"thread_start_us":12153,"threads_started":1}
I20260809 00:56:58.710003 4283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.685770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:46747 (local address 127.0.0.1:41250)
0809 00:56:58.698827 (+ 13057us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:58.698852 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:58.699019 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:58.700057 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:58.700061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:58.700090 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:58.700734 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.700757 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.702955 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.702961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.705976 (+ 3015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.705989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.707759 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.707768 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.707950 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.707982 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 00:56:58.709446 (+ 1464us) client_negotiation.cc:240] Negotiation successful
0809 00:56:58.709590 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12890,"mutex_wait_us":32,"thread_start_us":13357,"threads_started":1}
I20260809 00:56:58.711606 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.699957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.9.225.2:40169)
0809 00:56:58.701310 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:58.701319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:58.702115 (+ 796us) server_negotiation.cc:407] Connection header received
0809 00:56:58.702271 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:58.702279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:58.702509 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:58.702715 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:58.703693 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.705307 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.707692 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.709227 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.709389 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.709586 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:58.710269 (+ 683us) server_negotiation.cc:299] Negotiation successful
0809 00:56:58.710471 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1177,"thread_start_us":1002,"threads_started":1}
I20260809 00:56:58.711452 4288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.701361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:40169 (local address 127.0.0.1:58566)
0809 00:56:58.701973 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:58.701992 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:58.702124 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:58.702723 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:58.702733 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:58.702761 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:58.703467 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.703485 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.705467 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.705471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.707518 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.707534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.709780 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.709787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.710007 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.710033 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 00:56:58.710857 (+ 824us) client_negotiation.cc:240] Negotiation successful
0809 00:56:58.711009 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":26,"thread_start_us":305,"threads_started":1}
I20260809 00:56:58.713399 3756 tablet_service.cc:1467] Processing CreateTablet for tablet 4669da44974746829db8370eca9ec0bd (DEFAULT_TABLE table=test-table [id=994720d81119452d8e2b8128c4336cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:56:58.714112 4164 tablet_service.cc:1467] Processing CreateTablet for tablet 4669da44974746829db8370eca9ec0bd (DEFAULT_TABLE table=test-table [id=994720d81119452d8e2b8128c4336cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:56:58.715297 3756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4669da44974746829db8370eca9ec0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:58.715776 4164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4669da44974746829db8370eca9ec0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:58.715837 3957 tablet_service.cc:1467] Processing CreateTablet for tablet 4669da44974746829db8370eca9ec0bd (DEFAULT_TABLE table=test-table [id=994720d81119452d8e2b8128c4336cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:56:58.717324 3957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4669da44974746829db8370eca9ec0bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:56:58.731688 4293 tablet_bootstrap.cc:492] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596: Bootstrap starting.
I20260809 00:56:58.736191 4293 tablet_bootstrap.cc:654] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:58.737211 4297 tablet_bootstrap.cc:492] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f: Bootstrap starting.
I20260809 00:56:58.738250 4293 log.cc:824] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:58.739655 4295 tablet_bootstrap.cc:492] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3: Bootstrap starting.
I20260809 00:56:58.742779 4297 tablet_bootstrap.cc:654] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:58.743505 4295 tablet_bootstrap.cc:654] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3: Neither blocks nor log segments found. Creating new log.
I20260809 00:56:58.744381 4297 log.cc:824] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:58.744606 4295 log.cc:824] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3: Log is configured to *not* fsync() on all Append() calls
I20260809 00:56:58.744993 4293 tablet_bootstrap.cc:492] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596: No bootstrap required, opened a new log
I20260809 00:56:58.745451 4293 ts_tablet_manager.cc:1397] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260809 00:56:58.749989 4297 tablet_bootstrap.cc:492] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f: No bootstrap required, opened a new log
I20260809 00:56:58.750361 4297 ts_tablet_manager.cc:1397] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260809 00:56:58.760100 4295 tablet_bootstrap.cc:492] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3: No bootstrap required, opened a new log
I20260809 00:56:58.760586 4295 ts_tablet_manager.cc:1397] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.000s
I20260809 00:56:58.761456 4297 raft_consensus.cc:348] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } } peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } }
I20260809 00:56:58.762014 4297 raft_consensus.cc:374] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:58.762135 4297 raft_consensus.cc:729] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4371a34cdec043f49d7e0f01795a469f, State: Initialized, Role: FOLLOWER
I20260809 00:56:58.761978 4293 raft_consensus.cc:348] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } } peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } }
I20260809 00:56:58.762696 4293 raft_consensus.cc:374] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:58.762843 4293 raft_consensus.cc:729] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82a8a05b3794f9da2209fdfe0733596, State: Initialized, Role: FOLLOWER
I20260809 00:56:58.762910 4297 consensus_queue.cc:260] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } } peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } }
I20260809 00:56:58.763612 4293 consensus_queue.cc:260] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } } peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } }
I20260809 00:56:58.764461 4057 heartbeater.cc:503] Master 127.9.225.62:39741 was elected leader, sending a full tablet report...
I20260809 00:56:58.765045 4297 ts_tablet_manager.cc:1428] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f: Time spent starting tablet: real 0.015s user 0.006s sys 0.007s
I20260809 00:56:58.766050 4259 heartbeater.cc:503] Master 127.9.225.62:39741 was elected leader, sending a full tablet report...
I20260809 00:56:58.766902 4293 ts_tablet_manager.cc:1428] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596: Time spent starting tablet: real 0.021s user 0.013s sys 0.005s
I20260809 00:56:58.771961 4295 raft_consensus.cc:348] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } } peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } }
I20260809 00:56:58.772593 4295 raft_consensus.cc:374] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:56:58.772717 4295 raft_consensus.cc:729] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28867c48c5374d46adab0381ff985ba3, State: Initialized, Role: FOLLOWER
I20260809 00:56:58.773499 4295 consensus_queue.cc:260] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } } peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } }
I20260809 00:56:58.775276 4295 ts_tablet_manager.cc:1428] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3: Time spent starting tablet: real 0.015s user 0.005s sys 0.008s
W20260809 00:56:58.804152 4058 tablet.cc:2369] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:56:58.839936 4260 tablet.cc:2369] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:56:58.959892 4302 raft_consensus.cc:482] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:56:58.960233 4302 raft_consensus.cc:504] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } } peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } }
I20260809 00:56:58.962162 4302 leader_election.cc:288] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28867c48c5374d46adab0381ff985ba3 (127.9.225.1:44589), a82a8a05b3794f9da2209fdfe0733596 (127.9.225.3:46747)
I20260809 00:56:58.972754 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.962826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:45767 (local address 127.9.225.3:46747)
0809 00:56:58.963073 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:58.963081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:58.963788 (+ 707us) server_negotiation.cc:407] Connection header received
0809 00:56:58.963957 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:58.963963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:58.964039 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:58.964154 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:58.965097 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.966333 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.969875 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.971242 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.971410 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.971540 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:58.972300 (+ 760us) server_negotiation.cc:299] Negotiation successful
0809 00:56:58.972470 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 00:56:58.972839 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.962809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:53115 (local address 127.9.225.1:44589)
0809 00:56:58.963198 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:58.963204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:58.963821 (+ 617us) server_negotiation.cc:407] Connection header received
0809 00:56:58.963947 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:58.963953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:58.964039 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:58.964155 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:56:58.965097 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.966333 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.969900 (+ 3567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.971228 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.971411 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.971540 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:58.972300 (+ 760us) server_negotiation.cc:299] Negotiation successful
0809 00:56:58.972470 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260809 00:56:58.973037 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.962702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:46747 (local address 127.9.225.2:45767)
0809 00:56:58.963640 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:58.963658 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:58.963820 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:58.964222 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:58.964228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:58.964274 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:58.964890 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.964899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.966533 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.966537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.969607 (+ 3070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.969633 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.971625 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.971633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.971993 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.972105 (+ 112us) client_negotiation.cc:769] Sending connection context
0809 00:56:58.972220 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 00:56:58.972491 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":537,"spinlock_wait_cycles":64768,"thread_start_us":598,"threads_started":1}
I20260809 00:56:58.973453 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:58.962665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:44589 (local address 127.9.225.2:53115)
0809 00:56:58.963556 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0809 00:56:58.963579 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:56:58.963781 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:56:58.964222 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:56:58.964228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:56:58.964270 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:56:58.964832 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.964853 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.966525 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.966532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.969607 (+ 3075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:56:58.969633 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:56:58.971625 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:56:58.971633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:56:58.971989 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:58.972103 (+ 114us) client_negotiation.cc:769] Sending connection context
0809 00:56:58.972220 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 00:56:58.972491 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":325,"thread_start_us":411,"threads_started":1}
I20260809 00:56:58.974470 4197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4669da44974746829db8370eca9ec0bd" candidate_uuid: "4371a34cdec043f49d7e0f01795a469f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82a8a05b3794f9da2209fdfe0733596" is_pre_election: true
I20260809 00:56:58.974656 3786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4669da44974746829db8370eca9ec0bd" candidate_uuid: "4371a34cdec043f49d7e0f01795a469f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28867c48c5374d46adab0381ff985ba3" is_pre_election: true
I20260809 00:56:58.975134 4197 raft_consensus.cc:2454] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4371a34cdec043f49d7e0f01795a469f in term 0.
I20260809 00:56:58.975250 3786 raft_consensus.cc:2454] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4371a34cdec043f49d7e0f01795a469f in term 0.
I20260809 00:56:58.976042 3883 leader_election.cc:302] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28867c48c5374d46adab0381ff985ba3, 4371a34cdec043f49d7e0f01795a469f, a82a8a05b3794f9da2209fdfe0733596; no voters:
I20260809 00:56:58.976516 4302 raft_consensus.cc:2781] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:56:58.976671 4302 raft_consensus.cc:482] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:56:58.976795 4302 raft_consensus.cc:3037] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:58.979449 4302 raft_consensus.cc:504] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } } peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } }
I20260809 00:56:58.980327 4302 leader_election.cc:288] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [CANDIDATE]: Term 1 election: Requested vote from peers 28867c48c5374d46adab0381ff985ba3 (127.9.225.1:44589), a82a8a05b3794f9da2209fdfe0733596 (127.9.225.3:46747)
I20260809 00:56:58.981281 3786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4669da44974746829db8370eca9ec0bd" candidate_uuid: "4371a34cdec043f49d7e0f01795a469f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28867c48c5374d46adab0381ff985ba3"
I20260809 00:56:58.981281 4197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4669da44974746829db8370eca9ec0bd" candidate_uuid: "4371a34cdec043f49d7e0f01795a469f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82a8a05b3794f9da2209fdfe0733596"
I20260809 00:56:58.981820 4197 raft_consensus.cc:3037] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:58.981820 3786 raft_consensus.cc:3037] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:56:58.985894 4197 raft_consensus.cc:2454] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4371a34cdec043f49d7e0f01795a469f in term 1.
I20260809 00:56:58.985895 3786 raft_consensus.cc:2454] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4371a34cdec043f49d7e0f01795a469f in term 1.
I20260809 00:56:58.986569 3886 leader_election.cc:302] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28867c48c5374d46adab0381ff985ba3, 4371a34cdec043f49d7e0f01795a469f; no voters:
I20260809 00:56:58.987039 4302 raft_consensus.cc:2781] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:56:58.987660 4302 raft_consensus.cc:686] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 1 LEADER]: Becoming Leader. State: Replica: 4371a34cdec043f49d7e0f01795a469f, State: Running, Role: LEADER
I20260809 00:56:58.988258 4302 consensus_queue.cc:237] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } } peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } }
I20260809 00:56:58.993753 3570 catalog_manager.cc:5495] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4371a34cdec043f49d7e0f01795a469f (127.9.225.2). New cstate: current_term: 1 leader_uuid: "4371a34cdec043f49d7e0f01795a469f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4371a34cdec043f49d7e0f01795a469f" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 40169 } health_report { overall_health: HEALTHY } } }
W20260809 00:56:59.009626 3853 tablet.cc:2369] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:56:59.073271 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:59.066821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.9.225.2:40169)
0809 00:56:59.067024 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:59.067029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:59.067050 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:56:59.067322 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:59.067327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:59.067448 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:59.067634 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:56:59.068117 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:59.069001 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:59.070804 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:59.071305 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:59.071423 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:59.072053 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:56:59.072659 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:56:59.072795 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:59.072938 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 00:56:59.073024 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 00:56:59.083896 4197 raft_consensus.cc:1261] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596 [term 1 FOLLOWER]: Refusing update from remote peer 4371a34cdec043f49d7e0f01795a469f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:56:59.084458 3786 raft_consensus.cc:1261] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3 [term 1 FOLLOWER]: Refusing update from remote peer 4371a34cdec043f49d7e0f01795a469f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:56:59.085191 4312 consensus_queue.cc:1035] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82a8a05b3794f9da2209fdfe0733596" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:56:59.085743 4302 consensus_queue.cc:1035] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [LEADER]: Connected to new peer: Peer: permanent_uuid: "28867c48c5374d46adab0381ff985ba3" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:56:59.120572 3568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:56:59.120921 3568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:56:59.125315 3568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Sending DeleteTablet for 3 replicas of tablet 4669da44974746829db8370eca9ec0bd
I20260809 00:56:59.127478 3756 tablet_service.cc:1514] Processing DeleteTablet for tablet 4669da44974746829db8370eca9ec0bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:56:59 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:59474
I20260809 00:56:59.127310 3957 tablet_service.cc:1514] Processing DeleteTablet for tablet 4669da44974746829db8370eca9ec0bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:56:59 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:58566
I20260809 00:56:59.127454 4164 tablet_service.cc:1514] Processing DeleteTablet for tablet 4669da44974746829db8370eca9ec0bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:56:59 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:41250
I20260809 00:56:59.129118 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 3635
I20260809 00:56:59.138486 4334 tablet_replica.cc:330] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3: stopping tablet replica
I20260809 00:56:59.139247 4335 tablet_replica.cc:330] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596: stopping tablet replica
I20260809 00:56:59.139211 4334 raft_consensus.cc:2229] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:56:59.139686 4335 raft_consensus.cc:2229] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:56:59.139873 4334 raft_consensus.cc:2258] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:56:59.140172 4335 raft_consensus.cc:2258] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:56:59.142652 4334 ts_tablet_manager.cc:1905] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:56:59.144789 4336 tablet_replica.cc:330] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f: stopping tablet replica
I20260809 00:56:59.145494 4336 raft_consensus.cc:2229] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:56:59.146466 4336 raft_consensus.cc:2258] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:56:59.147933 4335 ts_tablet_manager.cc:1905] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:56:59.149030 4336 ts_tablet_manager.cc:1905] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:56:59.151328 4334 ts_tablet_manager.cc:1918] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:56:59.151564 4334 log.cc:1196] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/4669da44974746829db8370eca9ec0bd
I20260809 00:56:59.152199 4334 ts_tablet_manager.cc:1939] T 4669da44974746829db8370eca9ec0bd P 28867c48c5374d46adab0381ff985ba3: Deleting consensus metadata
I20260809 00:56:59.154536 3547 catalog_manager.cc:4841] TS 28867c48c5374d46adab0381ff985ba3 (127.9.225.1:44589): tablet 4669da44974746829db8370eca9ec0bd (table test-table [id=994720d81119452d8e2b8128c4336cef]) successfully deleted
I20260809 00:56:59.156766 4336 ts_tablet_manager.cc:1918] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:56:59.157004 4336 log.cc:1196] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/4669da44974746829db8370eca9ec0bd
I20260809 00:56:59.157742 4336 ts_tablet_manager.cc:1939] T 4669da44974746829db8370eca9ec0bd P 4371a34cdec043f49d7e0f01795a469f: Deleting consensus metadata
I20260809 00:56:59.160081 3546 catalog_manager.cc:4841] TS 4371a34cdec043f49d7e0f01795a469f (127.9.225.2:40169): tablet 4669da44974746829db8370eca9ec0bd (table test-table [id=994720d81119452d8e2b8128c4336cef]) successfully deleted
I20260809 00:56:59.162783 4335 ts_tablet_manager.cc:1918] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:56:59.162997 4335 log.cc:1196] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/4669da44974746829db8370eca9ec0bd
I20260809 00:56:59.163655 4335 ts_tablet_manager.cc:1939] T 4669da44974746829db8370eca9ec0bd P a82a8a05b3794f9da2209fdfe0733596: Deleting consensus metadata
I20260809 00:56:59.166081 3546 catalog_manager.cc:4841] TS a82a8a05b3794f9da2209fdfe0733596 (127.9.225.3:46747): tablet 4669da44974746829db8370eca9ec0bd (table test-table [id=994720d81119452d8e2b8128c4336cef]) successfully deleted
I20260809 00:56:59.172483 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:59.130576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.9.225.1:44589)
0809 00:56:59.130781 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:59.130788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:59.148991 (+ 18203us) server_negotiation.cc:407] Connection header received
0809 00:56:59.152075 (+ 3084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:59.152082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:59.152193 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:59.152282 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:59.157194 (+ 4912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:59.158485 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:59.160557 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:59.160996 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:59.161155 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:59.164771 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:59.164793 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:59.164810 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:59.164869 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:59.168894 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:59.169998 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:59.170012 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:59.170016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:59.170201 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:59.170660 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:59.170670 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:59.170673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:59.171106 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:59.171474 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:59.171820 (+ 346us) server_negotiation.cc:299] Negotiation successful
0809 00:56:59.172080 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 00:56:59.173758 3844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:59.278703 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 3635
I20260809 00:56:59.309931 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 3856
I20260809 00:56:59.330071 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:59.310920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.9.225.2:40169)
0809 00:56:59.311122 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:59.311127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:59.311186 (+ 59us) server_negotiation.cc:407] Connection header received
0809 00:56:59.311394 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:59.311399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:59.311490 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:59.311588 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:59.313701 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:59.314559 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:59.316864 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:59.317773 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:59.317913 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:59.321971 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:59.322002 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:59.322038 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:59.322102 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:59.326463 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:59.327639 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:59.327653 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:59.327655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:59.327864 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:59.328335 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:59.328347 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:59.328349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:59.328799 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:59.329093 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:59.329510 (+ 417us) server_negotiation.cc:299] Negotiation successful
0809 00:56:59.329709 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 00:56:59.330992 4046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:59.458698 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 3856
I20260809 00:56:59.486166 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 4063
I20260809 00:56:59.503222 4344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:56:59.487057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.9.225.3:46747)
0809 00:56:59.487594 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0809 00:56:59.487599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:56:59.487618 (+ 19us) server_negotiation.cc:407] Connection header received
0809 00:56:59.487677 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:56:59.487681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:56:59.487747 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:56:59.487856 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:56:59.489436 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:59.490186 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:59.491791 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:56:59.492279 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:56:59.492404 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:56:59.496029 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:56:59.496058 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:56:59.496081 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:56:59.496133 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:56:59.499447 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:59.500445 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:59.500456 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:59.500459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:59.500634 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:56:59.501130 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:56:59.501134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:56:59.501137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:56:59.501695 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:56:59.501980 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0809 00:56:59.502400 (+ 420us) server_negotiation.cc:299] Negotiation successful
0809 00:56:59.502630 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":261,"threads_started":1}
I20260809 00:56:59.503983 4254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:59.601037 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 4063
I20260809 00:56:59.645612 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 3524
I20260809 00:56:59.647548 3620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:56:59.769847 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 3524
I20260809 00:57:19.807638 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39741
--webserver_interface=127.9.225.62
--webserver_port=43759
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:57:20.204056 4350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:20.204422 4350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:20.204591 4350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:20.212584 4350 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 00:57:20.212723 4350 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 00:57:20.212808 4350 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 00:57:20.212901 4350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:57:20.212976 4350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:20.213094 4350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:20.213165 4350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:57:20.213264 4350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:57:20.224679 4350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=43759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:20.226361 4350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:20.227911 4350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:20.241997 4358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:20.242066 4361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:20.242066 4357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:20.242492 4350 server_base.cc:1034] running on GCE node
I20260809 00:57:20.243625 4350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:20.245275 4350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:20.246577 4350 hybrid_clock.cc:648] HybridClock initialized: now 1786237040246512 us; error 40 us; skew 500 ppm
I20260809 00:57:20.247033 4350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:20 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237040, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:20.250946 4350 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:57:20.252756 4350 webserver.cc:460] Webserver started at http://127.9.225.62:43759/ using document root <none> and password file <none>
I20260809 00:57:20.253665 4350 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:20.253784 4350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:20.260860 4350 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 00:57:20.264560 4372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:20.266498 4350 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260809 00:57:20.266700 4350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "3c6cac20e9c347b3a5c86bf76de4e24f"
format_stamp: "Formatted at 2026-08-09 00:56:32 on dist-test-slave-j2vl"
server_key: "c86c6c96f7ebd358bda03ec42ea7c85a"
server_key_iv: "98aa85a0430e6757e83f02dacd5aeb84"
server_key_version: "encryptionkey@0"
I20260809 00:57:20.267762 4350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:20.296589 4350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:20.301450 4350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:20.301731 4350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:20.323136 4350 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:39741
I20260809 00:57:20.323179 4449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:39741 every 8 connection(s)
I20260809 00:57:20.324877 4350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:57:20.332901 4450 sys_catalog.cc:263] Verifying existing consensus state
I20260809 00:57:20.333508 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 4350
I20260809 00:57:20.334460 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:44589
--local_ip_for_outbound_sockets=127.9.225.1
--tserver_master_addrs=127.9.225.62:39741
--webserver_port=36477
--webserver_interface=127.9.225.1
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:57:20.336884 4450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Bootstrap starting.
I20260809 00:57:20.359222 4450 log.cc:824] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:20.378173 4450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 00:57:20.378837 4450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Bootstrap complete.
I20260809 00:57:20.389571 4450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } }
I20260809 00:57:20.390692 4450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6cac20e9c347b3a5c86bf76de4e24f, State: Initialized, Role: FOLLOWER
I20260809 00:57:20.391530 4450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } }
I20260809 00:57:20.391820 4450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:57:20.391968 4450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:57:20.392169 4450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 2 FOLLOWER]: Advancing to term 3
I20260809 00:57:20.396711 4450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } }
I20260809 00:57:20.397392 4450 leader_election.cc:302] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c6cac20e9c347b3a5c86bf76de4e24f; no voters:
I20260809 00:57:20.398196 4450 leader_election.cc:288] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 00:57:20.398383 4455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 00:57:20.399979 4455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [term 3 LEADER]: Becoming Leader. State: Replica: 3c6cac20e9c347b3a5c86bf76de4e24f, State: Running, Role: LEADER
I20260809 00:57:20.400753 4455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } }
I20260809 00:57:20.401126 4450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:57:20.404002 4459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c6cac20e9c347b3a5c86bf76de4e24f. Latest consensus state: current_term: 3 leader_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } } }
I20260809 00:57:20.404541 4459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: This master's current role is: LEADER
I20260809 00:57:20.405160 4457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c6cac20e9c347b3a5c86bf76de4e24f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39741 } } }
I20260809 00:57:20.405611 4457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f [sys.catalog]: This master's current role is: LEADER
I20260809 00:57:20.409970 4464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:57:20.420794 4464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:57:20.422619 4464 catalog_manager.cc:1259] Loaded cluster ID: 9e1f2a5d7ba342cbb0d0b04512453327
I20260809 00:57:20.422839 4464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:57:20.427388 4464 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:57:20.429674 4464 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Loaded TSK: 0
I20260809 00:57:20.429961 4464 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c6cac20e9c347b3a5c86bf76de4e24f: Loaded TSK: 1
F20260809 00:57:20.431262 4464 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa16071c9ad google::LogMessage::Fail() at ??:0
@ 0x7fa16071f88a google::LogMessage::SendToLog() at ??:0
I20260809 00:57:20.439340 4480 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fa16071c4de google::LogMessage::Flush() at ??:0
@ 0x7fa16071da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa1644807d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa16446f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa17614b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa17614a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa1761d62a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa1761d60c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa1762205e9 std::function<>::operator()() at ??:0
@ 0x7fa17614ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa17614cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa1761d4610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa1761d4439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa163368345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa16336b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa16336b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa163346eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa1666206db start_thread at ??:0
@ 0x7fa15f5b661f clone at ??:0
W20260809 00:57:20.825052 4453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:20.825510 4453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:20.825702 4453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:20.833787 4453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:20.833956 4453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:20.834194 4453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:57:20.845114 4453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:44589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=36477
--tserver_master_addrs=127.9.225.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:20.846783 4453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:20.848280 4453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:20.863430 4527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:20.863392 4533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:20.863356 4530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:20.864661 4453 server_base.cc:1034] running on GCE node
I20260809 00:57:20.865494 4453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:20.866602 4453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:20.867865 4453 hybrid_clock.cc:648] HybridClock initialized: now 1786237040867800 us; error 57 us; skew 500 ppm
I20260809 00:57:20.868317 4453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:20 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237040, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:20.876597 4453 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:57:20.878767 4453 webserver.cc:460] Webserver started at http://127.9.225.1:36477/ using document root <none> and password file <none>
I20260809 00:57:20.879824 4453 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:20.879989 4453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:20.886972 4453 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 00:57:20.890507 4545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:20.892325 4453 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:57:20.892601 4453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "28867c48c5374d46adab0381ff985ba3"
format_stamp: "Formatted at 2026-08-09 00:56:33 on dist-test-slave-j2vl"
server_key: "f1c73520ec68f49b4e6f3f84ada3346c"
server_key_iv: "1762f653fe98838b501a1e3e301b6725"
server_key_version: "encryptionkey@0"
I20260809 00:57:20.893887 4453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:20.918354 4453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:20.923632 4453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:20.923934 4453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:20.925076 4453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:20.926698 4453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:20.926843 4453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:20.926962 4453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:20.927038 4453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:20.967001 4453 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:44589
I20260809 00:57:20.967103 4718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:44589 every 8 connection(s)
I20260809 00:57:20.968742 4453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:57:20.971618 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 4453
I20260809 00:57:20.972318 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:40169
--local_ip_for_outbound_sockets=127.9.225.2
--tserver_master_addrs=127.9.225.62:39741
--webserver_port=43065
--webserver_interface=127.9.225.2
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:57:20.975032 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:20.972510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:39245)
0809 00:57:20.973602 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:20.973937 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":425,"threads_started":1}
W20260809 00:57:20.975903 4720 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:39741 (0 consecutive failures): Network error: Failed to ping master at 127.9.225.62:39741: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
I20260809 00:57:20.976999 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:20.976598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:50977)
0809 00:57:20.976771 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:20.976858 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 00:57:20.978008 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:20.977620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:38165)
0809 00:57:20.977775 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:20.977883 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 00:57:20.978245 4720 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 00:57:21.359869 4728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:21.360212 4728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:21.360314 4728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:21.368798 4728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:21.368960 4728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:21.369148 4728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:57:21.380033 4728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:40169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=43065
--tserver_master_addrs=127.9.225.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:21.381652 4728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:21.383085 4728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:21.398509 4739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:21.398420 4742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:21.398466 4740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:21.400223 4728 server_base.cc:1034] running on GCE node
I20260809 00:57:21.401031 4728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:21.402285 4728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:21.403541 4728 hybrid_clock.cc:648] HybridClock initialized: now 1786237041403463 us; error 78 us; skew 500 ppm
I20260809 00:57:21.403947 4728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:21 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237041, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:21.407816 4728 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:57:21.409770 4728 webserver.cc:460] Webserver started at http://127.9.225.2:43065/ using document root <none> and password file <none>
I20260809 00:57:21.410681 4728 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:21.410859 4728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:21.417737 4728 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20260809 00:57:21.420980 4754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:21.422689 4728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 00:57:21.422947 4728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "4371a34cdec043f49d7e0f01795a469f"
format_stamp: "Formatted at 2026-08-09 00:56:34 on dist-test-slave-j2vl"
server_key: "d588176a98ee3c5aab7d019ebcd1006a"
server_key_iv: "25979e22386b69366c4988a1b356fca4"
server_key_version: "encryptionkey@0"
I20260809 00:57:21.424062 4728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:21.450532 4728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:21.455778 4728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:21.456127 4728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:21.457441 4728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:21.459321 4728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:21.459458 4728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:21.459601 4728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:21.459693 4728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:21.499003 4728 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:40169
I20260809 00:57:21.499035 4925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:40169 every 8 connection(s)
I20260809 00:57:21.500730 4728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:57:21.506424 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:21.504127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:48947)
0809 00:57:21.505072 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:21.505319 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":423,"threads_started":1}
W20260809 00:57:21.507238 4926 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:39741 (0 consecutive failures): Network error: Failed to ping master at 127.9.225.62:39741: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
I20260809 00:57:21.508293 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:21.507890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:41761)
0809 00:57:21.508078 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:21.508166 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 00:57:21.509333 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:21.508985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:35647)
0809 00:57:21.509130 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:21.509212 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 00:57:21.509678 4926 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 00:57:21.510298 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 4728
I20260809 00:57:21.510957 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:46747
--local_ip_for_outbound_sockets=127.9.225.3
--tserver_master_addrs=127.9.225.62:39741
--webserver_port=35955
--webserver_interface=127.9.225.3
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:57:21.893271 4930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:21.893802 4930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:21.893996 4930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:21.902233 4930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:21.902444 4930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:21.902655 4930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:57:21.913452 4930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:46747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=35955
--tserver_master_addrs=127.9.225.62:39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:21.915108 4930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:21.916649 4930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:21.934171 4939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:21.934255 4940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:21.934212 4942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:21.935240 4930 server_base.cc:1034] running on GCE node
I20260809 00:57:21.935986 4930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:21.937254 4930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:21.938585 4930 hybrid_clock.cc:648] HybridClock initialized: now 1786237041938539 us; error 36 us; skew 500 ppm
I20260809 00:57:21.939018 4930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:21 dist-test-slave-j2vl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237041, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:21.942962 4930 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:57:21.944808 4930 webserver.cc:460] Webserver started at http://127.9.225.3:35955/ using document root <none> and password file <none>
I20260809 00:57:21.945753 4930 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:21.945916 4930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:21.952270 4930 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:57:21.955644 4956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:21.957335 4930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 00:57:21.957628 4930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "a82a8a05b3794f9da2209fdfe0733596"
format_stamp: "Formatted at 2026-08-09 00:56:34 on dist-test-slave-j2vl"
server_key: "f12b031a780fda11cc4df99bd9f59a38"
server_key_iv: "06c5e632d77f5dc509ac5cd238391d32"
server_key_version: "encryptionkey@0"
I20260809 00:57:21.958739 4930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:21.980167 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:21.979068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:49893)
0809 00:57:21.979794 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:21.979900 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":430,"threads_started":1}
I20260809 00:57:21.980229 4930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:21.985211 4930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:21.985610 4930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:21.986847 4930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:21.988427 4930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:21.988528 4930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:21.988699 4930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:21.988804 4930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:22.026571 4930 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:46747
I20260809 00:57:22.026655 5141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:46747 every 8 connection(s)
I20260809 00:57:22.028466 4930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:57:22.034070 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:22.031957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:50863)
0809 00:57:22.032767 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:22.033000 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":314,"threads_started":1}
W20260809 00:57:22.034847 5142 heartbeater.cc:644] Failed to heartbeat to 127.9.225.62:39741 (0 consecutive failures): Network error: Failed to ping master at 127.9.225.62:39741: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
I20260809 00:57:22.035815 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:22.035424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:56655)
0809 00:57:22.035607 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:22.035689 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 00:57:22.036731 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:22.036360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:32815)
0809 00:57:22.036519 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:22.036601 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 00:57:22.037022 5142 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 00:57:22.037917 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 4930
I20260809 00:57:22.511482 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:22.510507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:56635)
0809 00:57:22.511090 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:22.511200 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":303,"threads_started":1}
I20260809 00:57:22.982409 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:22.981503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:51863)
0809 00:57:22.982095 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:22.982199 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":291,"threads_started":1}
I20260809 00:57:23.038750 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:23.037819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:37385)
0809 00:57:23.038367 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:23.038500 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":279,"threads_started":1}
I20260809 00:57:23.513809 5154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:23.512828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:45679)
0809 00:57:23.513491 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:23.513595 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":284,"threads_started":1}
I20260809 00:57:23.984474 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:23.983606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:56149)
0809 00:57:23.984167 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:23.984264 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":309,"threads_started":1}
I20260809 00:57:24.040684 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:24.039839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:46201)
0809 00:57:24.040398 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:24.040494 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":254,"threads_started":1}
I20260809 00:57:24.515799 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:24.514956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:47451)
0809 00:57:24.515493 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:24.515588 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":282,"threads_started":1}
I20260809 00:57:24.986585 5159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:24.985768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:53157)
0809 00:57:24.986294 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:24.986392 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":265,"threads_started":1}
I20260809 00:57:25.042636 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:25.041848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:57767)
0809 00:57:25.042347 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:25.042453 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":254,"threads_started":1}
I20260809 00:57:25.517997 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:25.517055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:33661)
0809 00:57:25.517681 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:25.517772 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":376,"threads_started":1}
I20260809 00:57:25.988700 5167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:25.987818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:45339)
0809 00:57:25.988389 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:25.988496 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":287,"threads_started":1}
I20260809 00:57:26.044663 5168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:26.043798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:60163)
0809 00:57:26.044334 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:26.044450 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":268,"threads_started":1}
I20260809 00:57:26.520212 5173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:26.519289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:44613)
0809 00:57:26.519899 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:26.520009 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":288,"threads_started":1}
I20260809 00:57:26.990850 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:26.990019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:48601)
0809 00:57:26.990548 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:26.990657 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
I20260809 00:57:27.046736 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:27.045915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:49891)
0809 00:57:27.046460 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:27.046552 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":271,"threads_started":1}
I20260809 00:57:27.522254 5178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:27.521397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:56831)
0809 00:57:27.521959 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:27.522069 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":257,"threads_started":1}
I20260809 00:57:27.993234 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:27.992125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:43661)
0809 00:57:27.992891 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:27.993003 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":452,"threads_started":1}
I20260809 00:57:28.048776 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:28.047969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:43467)
0809 00:57:28.048493 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:28.048598 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":272,"threads_started":1}
I20260809 00:57:28.524721 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:28.523651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:40917)
0809 00:57:28.524321 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:28.524443 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":336,"threads_started":1}
I20260809 00:57:28.995391 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:28.994455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:51641)
0809 00:57:28.995042 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:28.995145 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":299,"threads_started":1}
I20260809 00:57:29.051034 5185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:29.050090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:35885)
0809 00:57:29.050673 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:29.050773 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":270,"threads_started":1}
I20260809 00:57:29.526623 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:29.525786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:51987)
0809 00:57:29.526328 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:29.526421 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":300,"threads_started":1}
I20260809 00:57:29.997479 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:29.996490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:57721)
0809 00:57:29.997100 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:29.997192 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":305,"threads_started":1}
I20260809 00:57:30.052992 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:30.052114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:35105)
0809 00:57:30.052688 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:30.052803 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":306,"threads_started":1}
I20260809 00:57:30.528676 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:30.527736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:43479)
0809 00:57:30.528393 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:30.528485 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":280,"threads_started":1}
I20260809 00:57:31.000181 5190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:30.999013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:56343)
0809 00:57:30.999853 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:30.999972 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":485,"threads_started":1}
I20260809 00:57:31.055178 5191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:31.054201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:35233)
0809 00:57:31.054888 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:31.054987 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":363,"threads_started":1}
I20260809 00:57:31.530540 5193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:31.529739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:48721)
0809 00:57:31.530266 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:31.530358 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":268,"threads_started":1}
I20260809 00:57:32.002704 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:32.001522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:45427)
0809 00:57:32.002290 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:32.002431 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":397,"threads_started":1}
I20260809 00:57:32.057482 5195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:32.056505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:45765)
0809 00:57:32.057150 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:32.057232 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":424,"threads_started":1}
I20260809 00:57:32.532480 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:32.531634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:50853)
0809 00:57:32.532166 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:32.532270 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":291,"threads_started":1}
I20260809 00:57:33.005198 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:33.004109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:42301)
0809 00:57:33.004683 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:33.004815 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":317,"threads_started":1}
I20260809 00:57:33.059588 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:33.058691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:41527)
0809 00:57:33.059258 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:33.059365 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":286,"threads_started":1}
I20260809 00:57:33.534360 5203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:33.533563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:40617)
0809 00:57:33.534084 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:33.534177 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":283,"threads_started":1}
I20260809 00:57:34.007465 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:34.006446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:59375)
0809 00:57:34.007034 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:34.007240 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":273,"threads_started":1}
I20260809 00:57:34.061748 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:34.060956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:37535)
0809 00:57:34.061491 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:34.061577 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":238,"threads_started":1}
I20260809 00:57:34.536466 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:34.535521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:56483)
0809 00:57:34.536139 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:34.536254 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":305,"threads_started":1}
I20260809 00:57:35.009822 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:35.008753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:34383)
0809 00:57:35.009454 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:35.009575 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":320,"threads_started":1}
I20260809 00:57:35.063817 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:35.062858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:54601)
0809 00:57:35.063504 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:35.063617 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":293,"threads_started":1}
I20260809 00:57:35.538547 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:35.537722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:59715)
0809 00:57:35.538244 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:35.538341 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":265,"threads_started":1}
I20260809 00:57:36.012229 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:36.011338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:39635)
0809 00:57:36.011885 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:36.012029 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":283,"threads_started":1}
I20260809 00:57:36.065764 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:36.064859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:48755)
0809 00:57:36.065438 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:36.065535 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":253,"threads_started":1}
I20260809 00:57:36.540752 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:36.539885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.2:55515)
0809 00:57:36.540425 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:36.540551 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
I20260809 00:57:37.014400 5215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:37.013607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.1:36621)
0809 00:57:37.014130 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:37.014232 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":242,"threads_started":1}
I20260809 00:57:37.043493 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 4453
I20260809 00:57:37.065124 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:37.045236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.9.225.1:44589)
0809 00:57:37.045874 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:37.045890 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:37.045947 (+ 57us) server_negotiation.cc:407] Connection header received
0809 00:57:37.048089 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:37.048102 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:37.048601 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:37.048941 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:37.050680 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:37.052060 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:37.053499 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:37.054276 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:37.054431 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:37.057709 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:37.057721 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:37.057743 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:37.057800 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:37.061139 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:37.062125 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:37.062134 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:37.062143 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:37.062341 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:37.062726 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:37.062730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:37.062732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:37.063385 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:37.063723 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:37.064174 (+ 451us) server_negotiation.cc:299] Negotiation successful
0809 00:57:37.064732 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":286,"threads_started":1}
I20260809 00:57:37.067385 4713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:37.068110 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:37.067232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:39741 (local address 127.9.225.3:56151)
0809 00:57:37.067816 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:37.067922 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.62:39741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":312,"threads_started":1}
I20260809 00:57:37.156946 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 4453
I20260809 00:57:37.193920 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 4728
I20260809 00:57:37.214694 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:37.195067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.9.225.2:40169)
0809 00:57:37.195611 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:37.195627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:37.195669 (+ 42us) server_negotiation.cc:407] Connection header received
0809 00:57:37.197287 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:37.197298 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:37.197897 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:37.198315 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:37.200000 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:37.201467 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:37.202902 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:37.203702 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:37.203945 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:37.207112 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:37.207123 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:37.207143 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:37.207194 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:37.210725 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:37.211604 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:37.211612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:37.211619 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:37.211820 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:37.212248 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:37.212253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:37.212257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:37.212883 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:37.213226 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:37.213855 (+ 629us) server_negotiation.cc:299] Negotiation successful
0809 00:57:37.214362 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":233,"threads_started":1}
I20260809 00:57:37.216338 4918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:37.306147 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 4728
I20260809 00:57:37.334160 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 4930
I20260809 00:57:37.354943 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:37.335470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43604 (local address 127.9.225.3:46747)
0809 00:57:37.336220 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:37.336236 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:37.336287 (+ 51us) server_negotiation.cc:407] Connection header received
0809 00:57:37.338179 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:37.338193 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:37.338704 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:37.339067 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:37.340616 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:37.342056 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:37.343475 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:37.344279 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:37.344445 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:37.347455 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:37.347465 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:37.347485 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:37.347542 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:37.350938 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:37.351794 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:37.351802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:37.351810 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:37.352030 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:37.352527 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:37.352532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:37.352535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:37.353197 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:37.353523 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:37.354032 (+ 509us) server_negotiation.cc:299] Negotiation successful
0809 00:57:37.354541 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":353,"threads_started":1}
I20260809 00:57:37.356730 5134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:37.443133 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 4930
2026-08-09T00:57:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66640 ms)
[----------] 30 tests from SecurityITest (337991 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:57:37 dist-test-slave-j2vl krb5kdc[5229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:57:37 dist-test-slave-j2vl krb5kdc[5229](info): set up 2 sockets
Aug 09 00:57:37 dist-test-slave-j2vl krb5kdc[5229](info): commencing operation
krb5kdc: starting...
W20260809 00:57:38.847028 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.346s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:57:39 dist-test-slave-j2vl krb5kdc[5229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:57:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:57:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:57:39.212775 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:44441
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:44441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:57:39.598657 5265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:39.599020 5265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:39.599116 5265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:39.606887 5265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:57:39.607043 5265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:39.607091 5265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:39.607210 5265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:57:39.607295 5265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:57:39.618017 5265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:44441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:44441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:39.619545 5265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:39.620961 5265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:39.634861 5272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:39.634886 5276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:39.634912 5273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:39.635563 5265 server_base.cc:1034] running on GCE node
I20260809 00:57:39.636572 5265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:39.638371 5265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:39.639676 5265 hybrid_clock.cc:648] HybridClock initialized: now 1786237059639600 us; error 65 us; skew 500 ppm
I20260809 00:57:39.640154 5265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:39 dist-test-slave-j2vl krb5kdc[5229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237059, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:39.643759 5265 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:57:39.645692 5265 webserver.cc:460] Webserver started at http://127.9.225.62:46605/ using document root <none> and password file <none>
I20260809 00:57:39.646505 5265 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:39.646678 5265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:39.647042 5265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:39.650525 5265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "b3e24ef4b91247ca845081a9ab9086c9"
format_stamp: "Formatted at 2026-08-09 00:57:39 on dist-test-slave-j2vl"
server_key: "d74bb5f3b1ec8c991932b93cd30d3457"
server_key_iv: "5f007ff7c42b9b046fe2d7e5b798e53f"
server_key_version: "encryptionkey@0"
I20260809 00:57:39.651420 5265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "b3e24ef4b91247ca845081a9ab9086c9"
format_stamp: "Formatted at 2026-08-09 00:57:39 on dist-test-slave-j2vl"
server_key: "d74bb5f3b1ec8c991932b93cd30d3457"
server_key_iv: "5f007ff7c42b9b046fe2d7e5b798e53f"
server_key_version: "encryptionkey@0"
I20260809 00:57:39.657934 5265 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260809 00:57:39.662266 5287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:39.664142 5265 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:57:39.664407 5265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "b3e24ef4b91247ca845081a9ab9086c9"
format_stamp: "Formatted at 2026-08-09 00:57:39 on dist-test-slave-j2vl"
server_key: "d74bb5f3b1ec8c991932b93cd30d3457"
server_key_iv: "5f007ff7c42b9b046fe2d7e5b798e53f"
server_key_version: "encryptionkey@0"
I20260809 00:57:39.664669 5265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:39.691402 5265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:39.696659 5265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:39.697016 5265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:39.717489 5265 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:44441
I20260809 00:57:39.717509 5365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:44441 every 8 connection(s)
I20260809 00:57:39.719116 5265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:57:39.719910 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 5265
I20260809 00:57:39.720165 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:57:39.720870 10116 external_mini_cluster.cc:1428] Setting key fd619fd99bc6a6b333189316f9271e7d
I20260809 00:57:39.724668 5368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 00:57:39 dist-test-slave-j2vl krb5kdc[5229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:57:39.739498 5368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9: Bootstrap starting.
I20260809 00:57:39.743719 5368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:39.744983 5368 log.cc:824] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:39.747982 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:39.723391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.9.225.62:44441)
0809 00:57:39.724251 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:39.724263 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:39.724306 (+ 43us) server_negotiation.cc:407] Connection header received
0809 00:57:39.726259 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:39.726286 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:39.726748 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:39.727120 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:39.729261 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:39.731161 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:39.733279 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:39.734164 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:39.734405 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:39.739096 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:39.739106 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:39.739126 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:39.739176 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:39.742896 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:39.743889 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:39.743902 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:39.743913 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:39.744156 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:39.744646 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:39.744652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:39.744662 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:39.745468 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:39.745843 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:39.746377 (+ 534us) server_negotiation.cc:299] Negotiation successful
0809 00:57:39.746907 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":463,"threads_started":1}
I20260809 00:57:39.748657 5368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9: No bootstrap required, opened a new log
I20260809 00:57:39.757690 5368 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e24ef4b91247ca845081a9ab9086c9" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44441 } }
I20260809 00:57:39.758193 5368 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:39.758286 5368 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e24ef4b91247ca845081a9ab9086c9, State: Initialized, Role: FOLLOWER
I20260809 00:57:39.758919 5368 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e24ef4b91247ca845081a9ab9086c9" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44441 } }
I20260809 00:57:39.759148 5368 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:57:39.759262 5368 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:57:39.759416 5368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:39.762192 5368 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e24ef4b91247ca845081a9ab9086c9" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44441 } }
I20260809 00:57:39.762795 5368 leader_election.cc:302] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3e24ef4b91247ca845081a9ab9086c9; no voters:
I20260809 00:57:39.763453 5368 leader_election.cc:288] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:57:39.763645 5375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:57:39.764385 5375 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [term 1 LEADER]: Becoming Leader. State: Replica: b3e24ef4b91247ca845081a9ab9086c9, State: Running, Role: LEADER
I20260809 00:57:39.765038 5375 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e24ef4b91247ca845081a9ab9086c9" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44441 } }
I20260809 00:57:39.765969 5368 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:57:39.768822 5376 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3e24ef4b91247ca845081a9ab9086c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e24ef4b91247ca845081a9ab9086c9" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44441 } } }
I20260809 00:57:39.769475 5376 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [sys.catalog]: This master's current role is: LEADER
I20260809 00:57:39.769722 5377 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3e24ef4b91247ca845081a9ab9086c9. Latest consensus state: current_term: 1 leader_uuid: "b3e24ef4b91247ca845081a9ab9086c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e24ef4b91247ca845081a9ab9086c9" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 44441 } } }
I20260809 00:57:39.770515 5377 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9 [sys.catalog]: This master's current role is: LEADER
I20260809 00:57:39.774966 5389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:57:39.782241 5389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:57:39.793639 5389 catalog_manager.cc:1347] Generated new cluster ID: 91da63d5e6ff43f88611620e15dc8b82
I20260809 00:57:39.793840 5389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:57:39.805783 5389 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:57:39.808002 5389 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:57:39.820062 5389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9: Generated new TSK 0
I20260809 00:57:39.821089 5389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:57:39.909760 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44441
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:57:40.301285 5408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:40.301720 5408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:40.301928 5408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:40.309708 5408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:40.309890 5408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:40.310135 5408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:57:40.321694 5408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:40.323493 5408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:40.324952 5408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:40.340144 5415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:40.340281 5414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:40.340260 5418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:40.341096 5408 server_base.cc:1034] running on GCE node
I20260809 00:57:40.341900 5408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:40.343067 5408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:40.344349 5408 hybrid_clock.cc:648] HybridClock initialized: now 1786237060344286 us; error 50 us; skew 500 ppm
I20260809 00:57:40.344740 5408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:40 dist-test-slave-j2vl krb5kdc[5229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237060, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:40.348444 5408 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:57:40.350234 5408 webserver.cc:460] Webserver started at http://127.9.225.1:46045/ using document root <none> and password file <none>
I20260809 00:57:40.351073 5408 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:40.351249 5408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:40.351614 5408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:40.355177 5408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "965a9599aaa3432bb5224f44c146d594"
format_stamp: "Formatted at 2026-08-09 00:57:40 on dist-test-slave-j2vl"
server_key: "dc54b6988c1b74b89210539fee02a56e"
server_key_iv: "ee605b3ff62c61fb80240a91ff8d0656"
server_key_version: "encryptionkey@0"
I20260809 00:57:40.356422 5408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "965a9599aaa3432bb5224f44c146d594"
format_stamp: "Formatted at 2026-08-09 00:57:40 on dist-test-slave-j2vl"
server_key: "dc54b6988c1b74b89210539fee02a56e"
server_key_iv: "ee605b3ff62c61fb80240a91ff8d0656"
server_key_version: "encryptionkey@0"
I20260809 00:57:40.362991 5408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260809 00:57:40.367630 5427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:40.369724 5408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 00:57:40.369994 5408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "965a9599aaa3432bb5224f44c146d594"
format_stamp: "Formatted at 2026-08-09 00:57:40 on dist-test-slave-j2vl"
server_key: "dc54b6988c1b74b89210539fee02a56e"
server_key_iv: "ee605b3ff62c61fb80240a91ff8d0656"
server_key_version: "encryptionkey@0"
I20260809 00:57:40.370215 5408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:40.419284 5408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:40.423836 5408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:40.424108 5408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:40.425179 5408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:40.426743 5408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:40.426824 5408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:40.426921 5408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:40.426968 5408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:40.466310 5408 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:45685
I20260809 00:57:40.466347 5609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:45685 every 8 connection(s)
I20260809 00:57:40.468627 5408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:57:40.477459 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 5408
I20260809 00:57:40.477708 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:57:40.478430 10116 external_mini_cluster.cc:1428] Setting key f67e9cb2a6315e92b83a79b5c4288f44
Aug 09 00:57:40 dist-test-slave-j2vl krb5kdc[5229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237060, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:57:40.494391 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:40.473698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:49911 (local address 127.9.225.62:44441)
0809 00:57:40.475347 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:40.475354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:40.475384 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:57:40.476979 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:40.476986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:40.477096 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:40.477236 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:40.479267 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:40.480188 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:40.482185 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:40.482627 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:40.482801 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:40.487428 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:40.487434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:40.487438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:40.487481 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:40.490436 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:40.491384 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:40.491389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:40.491392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:40.491652 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:40.492251 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:40.492255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:40.492258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:40.492574 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:40.492751 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:40.493691 (+ 940us) server_negotiation.cc:299] Negotiation successful
0809 00:57:40.493992 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1514,"thread_start_us":1322,"threads_started":1}
I20260809 00:57:40.495719 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:40.473257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44441 (local address 127.9.225.1:49911)
0809 00:57:40.474313 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:40.474364 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:40.476640 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:40.477569 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:40.477580 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:40.478317 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:40.479054 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:40.479075 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:40.480518 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:40.480523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:40.481999 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:40.482014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:40.482918 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:40.482930 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:40.483204 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:40.484050 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:57:40.484079 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:40.487035 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:40.490652 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:40.490662 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:40.490681 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:40.491205 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:40.491825 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:40.491830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:40.491834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:40.492093 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:40.492725 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:40.492735 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:40.493178 (+ 443us) client_negotiation.cc:769] Sending connection context
0809 00:57:40.493556 (+ 378us) client_negotiation.cc:240] Negotiation successful
0809 00:57:40.494218 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":425,"threads_started":1}
I20260809 00:57:40.497645 5612 heartbeater.cc:347] Connected to a master server at 127.9.225.62:44441
I20260809 00:57:40.498047 5612 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:40.499161 5612 heartbeater.cc:511] Master 127.9.225.62:44441 requested a full tablet report, sending...
I20260809 00:57:40.501988 5310 ts_manager.cc:194] Registered new tserver with Master: 965a9599aaa3432bb5224f44c146d594 (127.9.225.1:45685)
I20260809 00:57:40.503600 5310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:49911
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:57:40.559530 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44441
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:57:40.954070 5622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:40.954420 5622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:40.954596 5622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:40.962611 5622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:40.962766 5622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:40.962950 5622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:57:40.974411 5622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:40.976138 5622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:40.977680 5622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:40.992657 5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:40.992616 5629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:40.992684 5632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:40.993574 5622 server_base.cc:1034] running on GCE node
I20260809 00:57:40.994330 5622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:40.995666 5622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:40.996977 5622 hybrid_clock.cc:648] HybridClock initialized: now 1786237060996929 us; error 71 us; skew 500 ppm
I20260809 00:57:40.997320 5622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:40 dist-test-slave-j2vl krb5kdc[5229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237060, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:41.006033 5622 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:57:41.008133 5622 webserver.cc:460] Webserver started at http://127.9.225.2:37033/ using document root <none> and password file <none>
I20260809 00:57:41.009024 5622 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:41.009186 5622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:41.009631 5622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:41.012981 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "50befc8a02534bf6a1d50f48c63376f4"
format_stamp: "Formatted at 2026-08-09 00:57:41 on dist-test-slave-j2vl"
server_key: "367c5f45df359bddbeae806aff468df1"
server_key_iv: "1ddab9f5d17860b1be7bb895b2d43638"
server_key_version: "encryptionkey@0"
I20260809 00:57:41.013952 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "50befc8a02534bf6a1d50f48c63376f4"
format_stamp: "Formatted at 2026-08-09 00:57:41 on dist-test-slave-j2vl"
server_key: "367c5f45df359bddbeae806aff468df1"
server_key_iv: "1ddab9f5d17860b1be7bb895b2d43638"
server_key_version: "encryptionkey@0"
I20260809 00:57:41.020378 5622 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:57:41.024564 5642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:41.026202 5622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 00:57:41.026439 5622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "50befc8a02534bf6a1d50f48c63376f4"
format_stamp: "Formatted at 2026-08-09 00:57:41 on dist-test-slave-j2vl"
server_key: "367c5f45df359bddbeae806aff468df1"
server_key_iv: "1ddab9f5d17860b1be7bb895b2d43638"
server_key_version: "encryptionkey@0"
I20260809 00:57:41.026690 5622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:41.065060 5622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:41.071156 5622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:41.071539 5622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:41.072718 5622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:41.075376 5622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:41.075501 5622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:41.075659 5622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:41.075802 5622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260809 00:57:41.111318 5622 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:43195
I20260809 00:57:41.111366 5808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:43195 every 8 connection(s)
I20260809 00:57:41.112833 5622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:57:41.118723 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 5622
I20260809 00:57:41.118978 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:57:41.119462 10116 external_mini_cluster.cc:1428] Setting key 1c56756ff51fb1f79484aa40d56ca7db
Aug 09 00:57:41 dist-test-slave-j2vl krb5kdc[5229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237060, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:57:41.139212 5812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:41.115921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:49741 (local address 127.9.225.62:44441)
0809 00:57:41.116412 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:41.116417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:41.117720 (+ 1303us) server_negotiation.cc:407] Connection header received
0809 00:57:41.119966 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:41.119974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:41.120083 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:41.120215 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:41.122581 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.123770 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.125579 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.126378 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.126536 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:41.130790 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:41.130801 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:41.130808 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:41.130863 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:41.134864 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:41.136001 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:41.136011 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:41.136017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:41.136202 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:41.136673 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:41.136681 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:41.136687 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:41.137133 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:41.137313 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:41.138446 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0809 00:57:41.138773 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":238,"threads_started":1}
I20260809 00:57:41.140619 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:41.116568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44441 (local address 127.9.225.2:49741)
0809 00:57:41.117524 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:41.117573 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:41.119604 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:41.120592 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:41.120602 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:41.121500 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:41.122362 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:41.122382 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.123981 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.123985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:41.125334 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:41.125418 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.126657 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.126663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:41.126913 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:41.127750 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:57:41.127781 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:41.130434 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:41.135073 (+ 4639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:41.135082 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:41.135099 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:41.135826 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:41.136324 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:41.136328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:41.136330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:41.136541 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:41.137296 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:41.137306 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:41.137936 (+ 630us) client_negotiation.cc:769] Sending connection context
0809 00:57:41.138316 (+ 380us) client_negotiation.cc:240] Negotiation successful
0809 00:57:41.139029 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":333,"threads_started":1}
I20260809 00:57:41.142417 5809 heartbeater.cc:347] Connected to a master server at 127.9.225.62:44441
I20260809 00:57:41.142796 5809 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:41.143635 5809 heartbeater.cc:511] Master 127.9.225.62:44441 requested a full tablet report, sending...
I20260809 00:57:41.145314 5310 ts_manager.cc:194] Registered new tserver with Master: 50befc8a02534bf6a1d50f48c63376f4 (127.9.225.2:43195)
I20260809 00:57:41.146279 5310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:49741
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:57:41.202250 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44441
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:57:41.508563 5612 heartbeater.cc:503] Master 127.9.225.62:44441 was elected leader, sending a full tablet report...
W20260809 00:57:41.590158 5818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:41.590516 5818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:41.590622 5818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:41.598366 5818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:41.598523 5818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:41.598645 5818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:57:41.609254 5818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:44441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:41.610706 5818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:41.612143 5818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:41.626911 5828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:41.626850 5826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:41.626941 5824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:41.627946 5818 server_base.cc:1034] running on GCE node
I20260809 00:57:41.628674 5818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:41.629958 5818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:41.631222 5818 hybrid_clock.cc:648] HybridClock initialized: now 1786237061631155 us; error 61 us; skew 500 ppm
I20260809 00:57:41.631726 5818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:41 dist-test-slave-j2vl krb5kdc[5229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237061, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:41.635401 5818 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:57:41.637192 5818 webserver.cc:460] Webserver started at http://127.9.225.3:39975/ using document root <none> and password file <none>
I20260809 00:57:41.638166 5818 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:41.638338 5818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:41.638706 5818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:41.642297 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "b2c5dd83be294eb780075dc078be8bc9"
format_stamp: "Formatted at 2026-08-09 00:57:41 on dist-test-slave-j2vl"
server_key: "3372d88449d9034cf4b6dfdbdf31354e"
server_key_iv: "7737a7e537f2de41b6b05368667f3117"
server_key_version: "encryptionkey@0"
I20260809 00:57:41.643198 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "b2c5dd83be294eb780075dc078be8bc9"
format_stamp: "Formatted at 2026-08-09 00:57:41 on dist-test-slave-j2vl"
server_key: "3372d88449d9034cf4b6dfdbdf31354e"
server_key_iv: "7737a7e537f2de41b6b05368667f3117"
server_key_version: "encryptionkey@0"
I20260809 00:57:41.649235 5818 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 00:57:41.653306 5838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:41.655087 5818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 00:57:41.655319 5818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "b2c5dd83be294eb780075dc078be8bc9"
format_stamp: "Formatted at 2026-08-09 00:57:41 on dist-test-slave-j2vl"
server_key: "3372d88449d9034cf4b6dfdbdf31354e"
server_key_iv: "7737a7e537f2de41b6b05368667f3117"
server_key_version: "encryptionkey@0"
I20260809 00:57:41.655568 5818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:41.670766 5818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:41.676143 5818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:41.676489 5818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:41.677663 5818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:41.679282 5818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:41.679396 5818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:41.679560 5818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:41.679644 5818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:41.720065 5818 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:34769
I20260809 00:57:41.720088 6008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:34769 every 8 connection(s)
I20260809 00:57:41.721671 5818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:57:41.730073 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 5818
I20260809 00:57:41.730314 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:57:41.730932 10116 external_mini_cluster.cc:1428] Setting key 1958f2ae63f32966de9cf5f1f51b1f64
Aug 09 00:57:41 dist-test-slave-j2vl krb5kdc[5229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237061, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:57:41.745600 6011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:41.724832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:57655 (local address 127.9.225.62:44441)
0809 00:57:41.725468 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:41.725473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:41.726574 (+ 1101us) server_negotiation.cc:407] Connection header received
0809 00:57:41.728643 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:41.728649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:41.728751 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:41.728884 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:41.731392 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.732641 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.734918 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.735331 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.735448 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:41.739303 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:41.739309 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:41.739313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:41.739351 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:41.742297 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:41.743175 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:41.743180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:41.743182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:41.743319 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:41.743744 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:41.743747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:41.743749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:41.744039 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:41.744171 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:41.744925 (+ 754us) server_negotiation.cc:299] Negotiation successful
0809 00:57:41.745127 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":297,"threads_started":1}
I20260809 00:57:41.747150 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:41.725544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:44441 (local address 127.9.225.3:57655)
0809 00:57:41.726378 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:41.726431 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:41.728301 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:41.729176 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:41.729185 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:41.730060 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:41.731106 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:41.731140 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.732878 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.732885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:41.734727 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:41.734748 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.735543 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.735548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:41.735768 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:41.736506 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:57:41.736531 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:41.739026 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:41.742472 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:41.742480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:41.742497 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:41.743022 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:41.743424 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:41.743427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:41.743430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:41.743636 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:41.744152 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:41.744159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:41.744553 (+ 394us) client_negotiation.cc:769] Sending connection context
0809 00:57:41.744866 (+ 313us) client_negotiation.cc:240] Negotiation successful
0809 00:57:41.745505 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":331,"threads_started":1}
I20260809 00:57:41.748791 6009 heartbeater.cc:347] Connected to a master server at 127.9.225.62:44441
I20260809 00:57:41.749135 6009 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:41.750022 6009 heartbeater.cc:511] Master 127.9.225.62:44441 requested a full tablet report, sending...
I20260809 00:57:41.751761 5310 ts_manager.cc:194] Registered new tserver with Master: b2c5dd83be294eb780075dc078be8bc9 (127.9.225.3:34769)
I20260809 00:57:41.752662 5310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:57655
I20260809 00:57:41.760767 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:57:41.781140 6011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:41.766336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.9.225.62:44441)
0809 00:57:41.766533 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:41.766538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:41.767072 (+ 534us) server_negotiation.cc:407] Connection header received
0809 00:57:41.767261 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:41.767266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:41.767341 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:41.767437 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:41.768954 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.770005 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.771525 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.771985 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.772094 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:41.775120 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:41.775126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:41.775130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:41.775176 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:41.778310 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:41.779162 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:41.779168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:41.779170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:41.779333 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:41.779702 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:41.779705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:41.779713 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:41.780085 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:41.780290 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:41.780662 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 00:57:41.780894 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 00:57:41.786083 5310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:57:41.794449 5310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:57:41.823009 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:41.810143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:34769 (local address 127.0.0.1:38826)
0809 00:57:41.812032 (+ 1889us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:41.812086 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:41.812360 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:41.814703 (+ 2343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:41.814708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:41.814755 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:41.815228 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:41.815243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.817646 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.817653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:41.819836 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:41.819852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.821826 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.821833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:41.822082 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:41.822148 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 00:57:41.822276 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 00:57:41.822501 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1527,"mutex_wait_us":126,"thread_start_us":733,"threads_started":1}
I20260809 00:57:41.823704 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:41.809830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.9.225.2:43195)
0809 00:57:41.813749 (+ 3919us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:41.813758 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:41.813789 (+ 31us) server_negotiation.cc:407] Connection header received
0809 00:57:41.813914 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:41.813922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:41.814215 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:41.814471 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:41.815412 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.817190 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.820653 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.822182 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.822333 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:41.822542 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:41.823192 (+ 650us) server_negotiation.cc:299] Negotiation successful
0809 00:57:41.823387 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3731,"thread_start_us":3434,"threads_started":1}
I20260809 00:57:41.823697 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:41.810735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:43195 (local address 127.0.0.1:48332)
0809 00:57:41.813050 (+ 2315us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:41.813069 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:41.813187 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:41.814501 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:41.814511 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:41.814544 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:41.815117 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:41.815132 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.817440 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.817447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:41.820430 (+ 2983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:41.820447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.822725 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.822732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:41.822950 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:41.822990 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 00:57:41.823104 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 00:57:41.823256 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2173,"mutex_wait_us":78,"thread_start_us":2019,"threads_started":1}
I20260809 00:57:41.824747 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:41.813159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:45685 (local address 127.0.0.1:60270)
0809 00:57:41.815771 (+ 2612us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:41.815786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:41.815941 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:41.816512 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:41.816517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:41.816541 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:41.816964 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:41.816978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.818878 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.818884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:41.821259 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:41.821274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.823952 (+ 2678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.823960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:41.824134 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:41.824166 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 00:57:41.824266 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 00:57:41.824425 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2499,"thread_start_us":2368,"threads_started":1}
I20260809 00:57:41.824896 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:41.813859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.9.225.1:45685)
0809 00:57:41.815314 (+ 1455us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:41.815326 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:41.815904 (+ 578us) server_negotiation.cc:407] Connection header received
0809 00:57:41.816118 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:41.816124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:41.816312 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:41.816522 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:41.817180 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.818719 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.821736 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.823261 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.823420 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:41.823668 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:41.824341 (+ 673us) server_negotiation.cc:299] Negotiation successful
0809 00:57:41.824525 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":1070,"threads_started":1}
I20260809 00:57:41.825172 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:41.810791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.9.225.3:34769)
0809 00:57:41.811821 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:41.811836 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:41.812251 (+ 415us) server_negotiation.cc:407] Connection header received
0809 00:57:41.812686 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:41.812699 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:41.814448 (+ 1749us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:41.815447 (+ 999us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:41.815746 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.817470 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.820019 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:41.821586 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:41.824065 (+ 2479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:41.824325 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:41.824406 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 00:57:41.824590 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":514,"threads_started":1}
I20260809 00:57:41.828693 5713 tablet_service.cc:1467] Processing CreateTablet for tablet 6f38f87a5bc14458a96467a0c450d445 (DEFAULT_TABLE table=test-table [id=84c22647140d4249ab27798da1227744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:57:41.829079 5508 tablet_service.cc:1467] Processing CreateTablet for tablet 6f38f87a5bc14458a96467a0c450d445 (DEFAULT_TABLE table=test-table [id=84c22647140d4249ab27798da1227744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:57:41.829123 5912 tablet_service.cc:1467] Processing CreateTablet for tablet 6f38f87a5bc14458a96467a0c450d445 (DEFAULT_TABLE table=test-table [id=84c22647140d4249ab27798da1227744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:57:41.830595 5912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f38f87a5bc14458a96467a0c450d445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:41.830603 5713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f38f87a5bc14458a96467a0c450d445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:41.830607 5508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f38f87a5bc14458a96467a0c450d445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:41.844549 6034 tablet_bootstrap.cc:492] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594: Bootstrap starting.
I20260809 00:57:41.847143 6038 tablet_bootstrap.cc:492] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4: Bootstrap starting.
I20260809 00:57:41.848104 6035 tablet_bootstrap.cc:492] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9: Bootstrap starting.
I20260809 00:57:41.849013 6034 tablet_bootstrap.cc:654] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:41.850615 6034 log.cc:824] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:41.852746 6038 tablet_bootstrap.cc:654] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:41.852823 6035 tablet_bootstrap.cc:654] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:41.854066 6038 log.cc:824] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:41.854517 6035 log.cc:824] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:41.857627 6034 tablet_bootstrap.cc:492] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594: No bootstrap required, opened a new log
I20260809 00:57:41.857827 6035 tablet_bootstrap.cc:492] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9: No bootstrap required, opened a new log
I20260809 00:57:41.858006 6034 ts_tablet_manager.cc:1397] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260809 00:57:41.858093 6035 ts_tablet_manager.cc:1397] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260809 00:57:41.859311 6038 tablet_bootstrap.cc:492] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4: No bootstrap required, opened a new log
I20260809 00:57:41.859681 6038 ts_tablet_manager.cc:1397] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260809 00:57:41.869192 6038 raft_consensus.cc:348] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50befc8a02534bf6a1d50f48c63376f4" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43195 } } peers { permanent_uuid: "b2c5dd83be294eb780075dc078be8bc9" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34769 } } peers { permanent_uuid: "965a9599aaa3432bb5224f44c146d594" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 45685 } }
I20260809 00:57:41.869886 6038 raft_consensus.cc:374] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:41.870031 6038 raft_consensus.cc:729] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50befc8a02534bf6a1d50f48c63376f4, State: Initialized, Role: FOLLOWER
I20260809 00:57:41.870836 6038 consensus_queue.cc:260] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50befc8a02534bf6a1d50f48c63376f4" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43195 } } peers { permanent_uuid: "b2c5dd83be294eb780075dc078be8bc9" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34769 } } peers { permanent_uuid: "965a9599aaa3432bb5224f44c146d594" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 45685 } }
I20260809 00:57:41.870811 6035 raft_consensus.cc:348] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50befc8a02534bf6a1d50f48c63376f4" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43195 } } peers { permanent_uuid: "b2c5dd83be294eb780075dc078be8bc9" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34769 } } peers { permanent_uuid: "965a9599aaa3432bb5224f44c146d594" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 45685 } }
I20260809 00:57:41.870811 6034 raft_consensus.cc:348] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50befc8a02534bf6a1d50f48c63376f4" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43195 } } peers { permanent_uuid: "b2c5dd83be294eb780075dc078be8bc9" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34769 } } peers { permanent_uuid: "965a9599aaa3432bb5224f44c146d594" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 45685 } }
I20260809 00:57:41.871335 6035 raft_consensus.cc:374] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:41.871335 6034 raft_consensus.cc:374] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:41.871454 6034 raft_consensus.cc:729] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965a9599aaa3432bb5224f44c146d594, State: Initialized, Role: FOLLOWER
I20260809 00:57:41.871455 6035 raft_consensus.cc:729] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c5dd83be294eb780075dc078be8bc9, State: Initialized, Role: FOLLOWER
I20260809 00:57:41.872125 6035 consensus_queue.cc:260] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50befc8a02534bf6a1d50f48c63376f4" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43195 } } peers { permanent_uuid: "b2c5dd83be294eb780075dc078be8bc9" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34769 } } peers { permanent_uuid: "965a9599aaa3432bb5224f44c146d594" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 45685 } }
I20260809 00:57:41.872121 6034 consensus_queue.cc:260] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50befc8a02534bf6a1d50f48c63376f4" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43195 } } peers { permanent_uuid: "b2c5dd83be294eb780075dc078be8bc9" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34769 } } peers { permanent_uuid: "965a9599aaa3432bb5224f44c146d594" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 45685 } }
I20260809 00:57:41.873636 6009 heartbeater.cc:503] Master 127.9.225.62:44441 was elected leader, sending a full tablet report...
I20260809 00:57:41.875051 6034 ts_tablet_manager.cc:1428] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20260809 00:57:41.876406 6035 ts_tablet_manager.cc:1428] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260809 00:57:41.876928 6038 ts_tablet_manager.cc:1428] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260809 00:57:41.877511 5809 heartbeater.cc:503] Master 127.9.225.62:44441 was elected leader, sending a full tablet report...
W20260809 00:57:41.972695 5615 tablet.cc:2369] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:57:41.973628 6010 tablet.cc:2369] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:57:42.106073 6051 raft_consensus.cc:482] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:57:42.106436 6051 raft_consensus.cc:504] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50befc8a02534bf6a1d50f48c63376f4" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43195 } } peers { permanent_uuid: "b2c5dd83be294eb780075dc078be8bc9" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34769 } } peers { permanent_uuid: "965a9599aaa3432bb5224f44c146d594" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 45685 } }
I20260809 00:57:42.108265 6051 leader_election.cc:288] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50befc8a02534bf6a1d50f48c63376f4 (127.9.225.2:43195), b2c5dd83be294eb780075dc078be8bc9 (127.9.225.3:34769)
W20260809 00:57:42.115695 5810 tablet.cc:2369] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:57:42.117399 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:42.108853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:45303 (local address 127.9.225.2:43195)
0809 00:57:42.109113 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:42.109123 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:42.109746 (+ 623us) server_negotiation.cc:407] Connection header received
0809 00:57:42.109987 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:42.109993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:42.110076 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:42.110183 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:42.111582 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.112722 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.114787 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.115993 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.116155 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:42.116282 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:42.116872 (+ 590us) server_negotiation.cc:299] Negotiation successful
0809 00:57:42.117062 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 00:57:42.117918 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:42.108853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:50195 (local address 127.9.225.3:34769)
0809 00:57:42.109300 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:42.109308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:42.109739 (+ 431us) server_negotiation.cc:407] Connection header received
0809 00:57:42.109987 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:42.109993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:42.110076 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:42.110183 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:42.111423 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.112621 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.115499 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.116818 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.116964 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:42.117070 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:42.117481 (+ 411us) server_negotiation.cc:299] Negotiation successful
0809 00:57:42.117636 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290}
I20260809 00:57:42.118022 6055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:42.108787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:43195 (local address 127.9.225.1:45303)
0809 00:57:42.109595 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:42.109622 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:42.109820 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:42.110245 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:42.110251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:42.110681 (+ 430us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:42.111374 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:42.111389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.112861 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.112871 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:42.114584 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:42.114600 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.116459 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.116468 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:42.116656 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:42.116697 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 00:57:42.117433 (+ 736us) client_negotiation.cc:240] Negotiation successful
0809 00:57:42.117600 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":69,"spinlock_wait_cycles":53888,"thread_start_us":532,"threads_started":1}
I20260809 00:57:42.118891 6053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:42.108662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:34769 (local address 127.9.225.1:50195)
0809 00:57:42.109595 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:42.109617 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:42.109820 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:42.110245 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:42.110251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:42.110681 (+ 430us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:42.111185 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:42.111205 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.112800 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.112807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:42.115239 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:42.115258 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.117049 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.117054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:42.117237 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:42.117270 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 00:57:42.118152 (+ 882us) client_negotiation.cc:240] Negotiation successful
0809 00:57:42.118351 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":36,"spinlock_wait_cycles":906112,"thread_start_us":442,"threads_started":1}
I20260809 00:57:42.119313 5743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f38f87a5bc14458a96467a0c450d445" candidate_uuid: "965a9599aaa3432bb5224f44c146d594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50befc8a02534bf6a1d50f48c63376f4" is_pre_election: true
I20260809 00:57:42.119968 5743 raft_consensus.cc:2454] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 965a9599aaa3432bb5224f44c146d594 in term 0.
I20260809 00:57:42.120086 5941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f38f87a5bc14458a96467a0c450d445" candidate_uuid: "965a9599aaa3432bb5224f44c146d594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c5dd83be294eb780075dc078be8bc9" is_pre_election: true
I20260809 00:57:42.120589 5941 raft_consensus.cc:2454] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 965a9599aaa3432bb5224f44c146d594 in term 0.
I20260809 00:57:42.120882 5433 leader_election.cc:302] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50befc8a02534bf6a1d50f48c63376f4, 965a9599aaa3432bb5224f44c146d594; no voters:
I20260809 00:57:42.121400 6051 raft_consensus.cc:2781] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:57:42.121613 6051 raft_consensus.cc:482] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:57:42.121755 6051 raft_consensus.cc:3037] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:42.124197 6051 raft_consensus.cc:504] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50befc8a02534bf6a1d50f48c63376f4" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43195 } } peers { permanent_uuid: "b2c5dd83be294eb780075dc078be8bc9" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34769 } } peers { permanent_uuid: "965a9599aaa3432bb5224f44c146d594" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 45685 } }
I20260809 00:57:42.124959 6051 leader_election.cc:288] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [CANDIDATE]: Term 1 election: Requested vote from peers 50befc8a02534bf6a1d50f48c63376f4 (127.9.225.2:43195), b2c5dd83be294eb780075dc078be8bc9 (127.9.225.3:34769)
I20260809 00:57:42.125586 5743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f38f87a5bc14458a96467a0c450d445" candidate_uuid: "965a9599aaa3432bb5224f44c146d594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50befc8a02534bf6a1d50f48c63376f4"
I20260809 00:57:42.125648 5941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f38f87a5bc14458a96467a0c450d445" candidate_uuid: "965a9599aaa3432bb5224f44c146d594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c5dd83be294eb780075dc078be8bc9"
I20260809 00:57:42.125849 5743 raft_consensus.cc:3037] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:42.125887 5941 raft_consensus.cc:3037] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:42.128517 5743 raft_consensus.cc:2454] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 965a9599aaa3432bb5224f44c146d594 in term 1.
I20260809 00:57:42.129170 5433 leader_election.cc:302] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50befc8a02534bf6a1d50f48c63376f4, 965a9599aaa3432bb5224f44c146d594; no voters:
I20260809 00:57:42.129282 5941 raft_consensus.cc:2454] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 965a9599aaa3432bb5224f44c146d594 in term 1.
I20260809 00:57:42.129675 6051 raft_consensus.cc:2781] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:57:42.130301 6051 raft_consensus.cc:686] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 1 LEADER]: Becoming Leader. State: Replica: 965a9599aaa3432bb5224f44c146d594, State: Running, Role: LEADER
I20260809 00:57:42.130939 6051 consensus_queue.cc:237] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50befc8a02534bf6a1d50f48c63376f4" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43195 } } peers { permanent_uuid: "b2c5dd83be294eb780075dc078be8bc9" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34769 } } peers { permanent_uuid: "965a9599aaa3432bb5224f44c146d594" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 45685 } }
I20260809 00:57:42.136904 5306 catalog_manager.cc:5495] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 965a9599aaa3432bb5224f44c146d594 (127.9.225.1). New cstate: current_term: 1 leader_uuid: "965a9599aaa3432bb5224f44c146d594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50befc8a02534bf6a1d50f48c63376f4" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2c5dd83be294eb780075dc078be8bc9" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "965a9599aaa3432bb5224f44c146d594" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 45685 } health_report { overall_health: HEALTHY } } }
I20260809 00:57:42.197820 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:42.191547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.9.225.1:45685)
0809 00:57:42.191790 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:42.191798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:42.191923 (+ 125us) server_negotiation.cc:407] Connection header received
0809 00:57:42.192203 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:42.192208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:42.192295 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:42.192481 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:57:42.192893 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.193670 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.195315 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.195739 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.195890 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:42.196522 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:57:42.197131 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:57:42.197333 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:42.197425 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 00:57:42.197523 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 00:57:42.207239 5941 raft_consensus.cc:1261] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9 [term 1 FOLLOWER]: Refusing update from remote peer 965a9599aaa3432bb5224f44c146d594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:57:42.207237 5743 raft_consensus.cc:1261] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4 [term 1 FOLLOWER]: Refusing update from remote peer 965a9599aaa3432bb5224f44c146d594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:57:42.208438 6056 consensus_queue.cc:1035] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50befc8a02534bf6a1d50f48c63376f4" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 43195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:57:42.208938 6051 consensus_queue.cc:1035] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2c5dd83be294eb780075dc078be8bc9" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:57:42.222254 6061 mvcc.cc:205] Tried to move back new op lower bound from 7316427006788956160 to 7316427006493466624. Current Snapshot: MvccSnapshot[applied={T|T < 7316427006788956160}]
I20260809 00:57:42.223984 6063 mvcc.cc:205] Tried to move back new op lower bound from 7316427006788956160 to 7316427006493466624. Current Snapshot: MvccSnapshot[applied={T|T < 7316427006788956160}]
I20260809 00:57:42.238238 5306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:57:42.238610 5306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:57:42.242769 5306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3e24ef4b91247ca845081a9ab9086c9: Sending DeleteTablet for 3 replicas of tablet 6f38f87a5bc14458a96467a0c450d445
I20260809 00:57:42.244390 5912 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f38f87a5bc14458a96467a0c450d445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:57:42 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:38826
I20260809 00:57:42.244386 5713 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f38f87a5bc14458a96467a0c450d445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:57:42 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:48332
I20260809 00:57:42.244578 5508 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f38f87a5bc14458a96467a0c450d445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:57:42 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:60270
I20260809 00:57:42.251161 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 5408
I20260809 00:57:42.256953 6076 tablet_replica.cc:330] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594: stopping tablet replica
I20260809 00:57:42.257138 6074 tablet_replica.cc:330] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9: stopping tablet replica
I20260809 00:57:42.257763 6074 raft_consensus.cc:2229] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:57:42.258401 6076 raft_consensus.cc:2229] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:57:42.258531 6074 raft_consensus.cc:2258] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:57:42.259752 6075 tablet_replica.cc:330] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4: stopping tablet replica
I20260809 00:57:42.260633 6076 raft_consensus.cc:2258] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:57:42.261840 6075 raft_consensus.cc:2229] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:57:42.262452 6075 raft_consensus.cc:2258] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:57:42.263154 6074 ts_tablet_manager.cc:1905] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:57:42.263767 6076 ts_tablet_manager.cc:1905] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 00:57:42 dist-test-slave-j2vl krb5kdc[5229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:57:42.269459 6075 ts_tablet_manager.cc:1905] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:57:42.273413 6076 ts_tablet_manager.cc:1918] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:57:42.273741 6076 log.cc:1196] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/6f38f87a5bc14458a96467a0c450d445
I20260809 00:57:42.274485 6076 ts_tablet_manager.cc:1939] T 6f38f87a5bc14458a96467a0c450d445 P 965a9599aaa3432bb5224f44c146d594: Deleting consensus metadata
I20260809 00:57:42.276854 5291 catalog_manager.cc:4841] TS 965a9599aaa3432bb5224f44c146d594 (127.9.225.1:45685): tablet 6f38f87a5bc14458a96467a0c450d445 (table test-table [id=84c22647140d4249ab27798da1227744]) successfully deleted
I20260809 00:57:42.278611 6074 ts_tablet_manager.cc:1918] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:57:42.278834 6074 log.cc:1196] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/6f38f87a5bc14458a96467a0c450d445
I20260809 00:57:42.279413 6074 ts_tablet_manager.cc:1939] T 6f38f87a5bc14458a96467a0c450d445 P b2c5dd83be294eb780075dc078be8bc9: Deleting consensus metadata
I20260809 00:57:42.279520 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:42.254168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.9.225.1:45685)
0809 00:57:42.254391 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:42.254399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:42.254654 (+ 255us) server_negotiation.cc:407] Connection header received
0809 00:57:42.255876 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:42.255886 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:42.256015 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:42.256137 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:42.258648 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.260036 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.262470 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.263435 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.263553 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:42.270143 (+ 6590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:42.270168 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:42.270193 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:42.270257 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:42.275084 (+ 4827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:42.276363 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:42.276386 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:42.276392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:42.276602 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:42.277144 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:42.277149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:42.277152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:42.277714 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:42.278058 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:42.278764 (+ 706us) server_negotiation.cc:299] Negotiation successful
0809 00:57:42.279090 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 00:57:42.281373 6075 ts_tablet_manager.cc:1918] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:57:42.281574 5604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:42.281719 6075 log.cc:1196] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/6f38f87a5bc14458a96467a0c450d445
I20260809 00:57:42.281952 5289 catalog_manager.cc:4841] TS b2c5dd83be294eb780075dc078be8bc9 (127.9.225.3:34769): tablet 6f38f87a5bc14458a96467a0c450d445 (table test-table [id=84c22647140d4249ab27798da1227744]) successfully deleted
I20260809 00:57:42.282624 6075 ts_tablet_manager.cc:1939] T 6f38f87a5bc14458a96467a0c450d445 P 50befc8a02534bf6a1d50f48c63376f4: Deleting consensus metadata
I20260809 00:57:42.285226 5291 catalog_manager.cc:4841] TS 50befc8a02534bf6a1d50f48c63376f4 (127.9.225.2:43195): tablet 6f38f87a5bc14458a96467a0c450d445 (table test-table [id=84c22647140d4249ab27798da1227744]) successfully deleted
I20260809 00:57:42.387507 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 5408
I20260809 00:57:42.416249 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 5622
Aug 09 00:57:42 dist-test-slave-j2vl krb5kdc[5229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:57:42.433835 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:42.417146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.9.225.2:43195)
0809 00:57:42.417374 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:42.417381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:42.417460 (+ 79us) server_negotiation.cc:407] Connection header received
0809 00:57:42.417685 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:42.417690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:42.417776 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:42.417891 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:42.419329 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.420116 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.421633 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.422092 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.422391 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:42.427428 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:42.427458 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:42.427480 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:42.427528 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:42.430641 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:42.431647 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:42.431656 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:42.431659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:42.431820 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:42.432191 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:42.432195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:42.432197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:42.432648 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:42.432908 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:42.433225 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 00:57:42.433529 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 00:57:42.434607 5802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:42.533488 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 5622
I20260809 00:57:42.558248 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 5818
Aug 09 00:57:42 dist-test-slave-j2vl krb5kdc[5229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:57:42.576898 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:42.559548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.9.225.3:34769)
0809 00:57:42.559779 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:42.559784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:42.559805 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:57:42.559904 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:42.559908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:42.559979 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:42.560080 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:42.561621 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.562378 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.563988 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:42.564413 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:42.564524 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:42.569753 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:42.569772 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:42.569794 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:42.569868 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:42.573100 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:42.574357 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:42.574367 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:42.574370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:42.574560 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:42.575070 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:42.575074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:42.575077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:42.575669 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:42.575947 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:42.576385 (+ 438us) server_negotiation.cc:299] Negotiation successful
0809 00:57:42.576591 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260809 00:57:42.577814 6002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:42.672561 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 5818
I20260809 00:57:42.702683 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 5265
I20260809 00:57:42.704020 5358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:42.821681 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 5265
2026-08-09T00:57:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5385 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T00:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:57:42Z Disabled control of system clock
I20260809 00:57:42.940450 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:42219
--webserver_interface=127.9.225.62
--webserver_port=0
--builtin_ntp_servers=127.9.225.20:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 00:57:43.338320 6091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:43.338683 6091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:43.338884 6091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:43.346931 6091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:57:43.347087 6091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:43.347184 6091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:43.347307 6091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:57:43.347376 6091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:57:43.357932 6091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:42219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:43.359532 6091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:43.361743 6091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:43.375468 6104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:43.375751 6103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:43.375499 6108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:43.376528 6091 server_base.cc:1034] running on GCE node
I20260809 00:57:43.377704 6091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:43.379503 6091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:43.380810 6091 hybrid_clock.cc:648] HybridClock initialized: now 1786237063380726 us; error 61 us; skew 500 ppm
I20260809 00:57:43.381234 6091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:57:43.384152 6091 webserver.cc:460] Webserver started at http://127.9.225.62:35433/ using document root <none> and password file <none>
I20260809 00:57:43.385046 6091 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:43.385195 6091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:43.385622 6091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:43.389112 6091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "8cbd2af3cf444afd8053ae62e5cc5e3b"
format_stamp: "Formatted at 2026-08-09 00:57:43 on dist-test-slave-j2vl"
server_key: "caef1af94a2af5545eb6c0c595492088"
server_key_iv: "f520af9c0267670a74f7adf46f9c8647"
server_key_version: "encryptionkey@0"
I20260809 00:57:43.390149 6091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "8cbd2af3cf444afd8053ae62e5cc5e3b"
format_stamp: "Formatted at 2026-08-09 00:57:43 on dist-test-slave-j2vl"
server_key: "caef1af94a2af5545eb6c0c595492088"
server_key_iv: "f520af9c0267670a74f7adf46f9c8647"
server_key_version: "encryptionkey@0"
I20260809 00:57:43.396500 6091 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260809 00:57:43.400969 6117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:43.402922 6091 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260809 00:57:43.403177 6091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "8cbd2af3cf444afd8053ae62e5cc5e3b"
format_stamp: "Formatted at 2026-08-09 00:57:43 on dist-test-slave-j2vl"
server_key: "caef1af94a2af5545eb6c0c595492088"
server_key_iv: "f520af9c0267670a74f7adf46f9c8647"
server_key_version: "encryptionkey@0"
I20260809 00:57:43.403462 6091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:43.420871 6091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:43.421983 6091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:43.422329 6091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:43.443388 6091 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:42219
I20260809 00:57:43.443408 6197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:42219 every 8 connection(s)
I20260809 00:57:43.445003 6091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:57:43.447697 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 6091
I20260809 00:57:43.447919 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:57:43.448539 10116 external_mini_cluster.cc:1428] Setting key e0c530d36000df7e749ceaefbf630aa2
I20260809 00:57:43.450306 6198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:43.465039 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:43.450826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.9.225.62:42219)
0809 00:57:43.451983 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:43.451995 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:43.452044 (+ 49us) server_negotiation.cc:407] Connection header received
0809 00:57:43.454001 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:43.454031 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:43.454602 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:43.455133 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:43.455813 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:43.457304 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:43.458898 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:43.459831 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:43.460087 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:43.461633 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:43.461645 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:43.461671 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:43.462577 (+ 906us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:43.462688 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:43.462720 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:43.462872 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:43.463127 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 00:57:43.463543 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":445,"threads_started":1}
I20260809 00:57:43.465512 6198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b: Bootstrap starting.
I20260809 00:57:43.469551 6198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:43.470882 6198 log.cc:824] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:43.474393 6198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b: No bootstrap required, opened a new log
I20260809 00:57:43.483310 6198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbd2af3cf444afd8053ae62e5cc5e3b" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42219 } }
I20260809 00:57:43.483793 6198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:43.483958 6198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cbd2af3cf444afd8053ae62e5cc5e3b, State: Initialized, Role: FOLLOWER
I20260809 00:57:43.484722 6198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbd2af3cf444afd8053ae62e5cc5e3b" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42219 } }
I20260809 00:57:43.484994 6198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:57:43.485148 6198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:57:43.485406 6198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:43.488210 6198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbd2af3cf444afd8053ae62e5cc5e3b" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42219 } }
I20260809 00:57:43.488828 6198 leader_election.cc:302] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cbd2af3cf444afd8053ae62e5cc5e3b; no voters:
I20260809 00:57:43.489585 6198 leader_election.cc:288] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:57:43.489779 6204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:57:43.490577 6204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [term 1 LEADER]: Becoming Leader. State: Replica: 8cbd2af3cf444afd8053ae62e5cc5e3b, State: Running, Role: LEADER
I20260809 00:57:43.491235 6204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbd2af3cf444afd8053ae62e5cc5e3b" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42219 } }
I20260809 00:57:43.492020 6198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:57:43.495215 6207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cbd2af3cf444afd8053ae62e5cc5e3b. Latest consensus state: current_term: 1 leader_uuid: "8cbd2af3cf444afd8053ae62e5cc5e3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbd2af3cf444afd8053ae62e5cc5e3b" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42219 } } }
I20260809 00:57:43.495215 6205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cbd2af3cf444afd8053ae62e5cc5e3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbd2af3cf444afd8053ae62e5cc5e3b" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42219 } } }
I20260809 00:57:43.495748 6207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [sys.catalog]: This master's current role is: LEADER
I20260809 00:57:43.495754 6205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b [sys.catalog]: This master's current role is: LEADER
I20260809 00:57:43.497623 6216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:57:43.502666 6216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:57:43.513818 6216 catalog_manager.cc:1347] Generated new cluster ID: 8852fd26424c4656b26345b8f4c9e3f2
I20260809 00:57:43.513995 6216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:57:43.550218 6216 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:57:43.551882 6216 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:57:43.564621 6216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b: Generated new TSK 0
I20260809 00:57:43.565904 6216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:57:43.581308 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42219
--builtin_ntp_servers=127.9.225.20:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 00:57:43.988453 6234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:43.988792 6234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:43.988986 6234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:43.997011 6234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:43.997192 6234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:43.997442 6234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:57:44.007694 6234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:44.009404 6234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:44.011438 6234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:44.026477 6240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:44.026450 6245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:44.026420 6241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:44.027472 6234 server_base.cc:1034] running on GCE node
I20260809 00:57:44.028131 6234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:44.029088 6234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:44.030309 6234 hybrid_clock.cc:648] HybridClock initialized: now 1786237064030252 us; error 55 us; skew 500 ppm
I20260809 00:57:44.030690 6234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:57:44.033211 6234 webserver.cc:460] Webserver started at http://127.9.225.1:45783/ using document root <none> and password file <none>
I20260809 00:57:44.034030 6234 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:44.034153 6234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:44.034494 6234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:44.037734 6234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "883311029c2e48e5bab5a19f13ecb91a"
format_stamp: "Formatted at 2026-08-09 00:57:44 on dist-test-slave-j2vl"
server_key: "287277132b56035602c6c61c68253966"
server_key_iv: "e7870ffaa97e0be9156a009b027d53db"
server_key_version: "encryptionkey@0"
I20260809 00:57:44.038661 6234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "883311029c2e48e5bab5a19f13ecb91a"
format_stamp: "Formatted at 2026-08-09 00:57:44 on dist-test-slave-j2vl"
server_key: "287277132b56035602c6c61c68253966"
server_key_iv: "e7870ffaa97e0be9156a009b027d53db"
server_key_version: "encryptionkey@0"
I20260809 00:57:44.044480 6234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260809 00:57:44.049479 6256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:44.051532 6234 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260809 00:57:44.051782 6234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "883311029c2e48e5bab5a19f13ecb91a"
format_stamp: "Formatted at 2026-08-09 00:57:44 on dist-test-slave-j2vl"
server_key: "287277132b56035602c6c61c68253966"
server_key_iv: "e7870ffaa97e0be9156a009b027d53db"
server_key_version: "encryptionkey@0"
I20260809 00:57:44.052130 6234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:44.100013 6234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:44.101052 6234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:44.101420 6234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:44.102700 6234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:44.104493 6234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:44.104591 6234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:44.104686 6234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:44.104743 6234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:44.141781 6234 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:39817
I20260809 00:57:44.141835 6455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:39817 every 8 connection(s)
I20260809 00:57:44.143323 6234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:57:44.150939 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 6234
I20260809 00:57:44.151147 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:57:44.151755 10116 external_mini_cluster.cc:1428] Setting key 02585d39017c297c28ecec36420f134c
I20260809 00:57:44.155248 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42219
--builtin_ntp_servers=127.9.225.20:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 00:57:44.159096 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:44.146443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:57483 (local address 127.9.225.62:42219)
0809 00:57:44.146949 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:44.146956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:44.148296 (+ 1340us) server_negotiation.cc:407] Connection header received
0809 00:57:44.150314 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:44.150319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:44.150405 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:44.150519 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:44.152182 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:44.153098 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:44.155209 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:44.155994 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:44.156163 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:44.157952 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:44.157961 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:44.157969 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:44.157999 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:44.158056 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:44.158076 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:44.158181 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:44.158490 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 00:57:44.158624 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":235,"threads_started":1}
I20260809 00:57:44.160085 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:44.147102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:42219 (local address 127.9.225.1:57483)
0809 00:57:44.148091 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:44.148140 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:44.150003 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:44.150853 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:44.150867 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:44.151002 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:44.151934 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:44.151964 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:44.153307 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:44.153312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:44.155021 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:44.155037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:44.156382 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:44.156390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:44.156645 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:44.157560 (+ 915us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:57:44.157562 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:44.157611 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:57:44.157709 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:44.158211 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:44.158222 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:44.158231 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 00:57:44.158425 (+ 194us) client_negotiation.cc:240] Negotiation successful
0809 00:57:44.158792 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":314,"threads_started":1}
I20260809 00:57:44.162276 6456 heartbeater.cc:347] Connected to a master server at 127.9.225.62:42219
I20260809 00:57:44.162638 6456 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:44.163553 6456 heartbeater.cc:511] Master 127.9.225.62:42219 requested a full tablet report, sending...
I20260809 00:57:44.166054 6146 ts_manager.cc:194] Registered new tserver with Master: 883311029c2e48e5bab5a19f13ecb91a (127.9.225.1:39817)
I20260809 00:57:44.167796 6146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.225.1:57483
W20260809 00:57:44.948064 6463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:44.948488 6463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:44.948657 6463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:44.964788 6463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:44.965004 6463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:44.965219 6463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:57:44.978940 6463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:44.981784 6463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:44.986263 6463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:45.019214 6473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:45.019145 6470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:45.019263 6475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:45.025626 6463 server_base.cc:1034] running on GCE node
I20260809 00:57:45.026541 6463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:45.029557 6463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:45.033492 6463 hybrid_clock.cc:648] HybridClock initialized: now 1786237065033205 us; error 162 us; skew 500 ppm
I20260809 00:57:45.034412 6463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:57:45.038833 6463 webserver.cc:460] Webserver started at http://127.9.225.2:39721/ using document root <none> and password file <none>
I20260809 00:57:45.040400 6463 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:45.040589 6463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:45.041108 6463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:45.048002 6463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "06dbdbc4c0f14796a91bea00ccf354c6"
format_stamp: "Formatted at 2026-08-09 00:57:45 on dist-test-slave-j2vl"
server_key: "c4f5f973de654578f55083966c7ce512"
server_key_iv: "826d0331eb796c9016cfb09cf7a40a91"
server_key_version: "encryptionkey@0"
I20260809 00:57:45.050200 6463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "06dbdbc4c0f14796a91bea00ccf354c6"
format_stamp: "Formatted at 2026-08-09 00:57:45 on dist-test-slave-j2vl"
server_key: "c4f5f973de654578f55083966c7ce512"
server_key_iv: "826d0331eb796c9016cfb09cf7a40a91"
server_key_version: "encryptionkey@0"
I20260809 00:57:45.065145 6463 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.013s sys 0.000s
I20260809 00:57:45.074659 6486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:45.076543 6463 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260809 00:57:45.076803 6463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "06dbdbc4c0f14796a91bea00ccf354c6"
format_stamp: "Formatted at 2026-08-09 00:57:45 on dist-test-slave-j2vl"
server_key: "c4f5f973de654578f55083966c7ce512"
server_key_iv: "826d0331eb796c9016cfb09cf7a40a91"
server_key_version: "encryptionkey@0"
I20260809 00:57:45.077525 6463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:45.111124 6463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:45.112124 6463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:45.112409 6463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:45.113682 6463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:45.116216 6463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:45.116349 6463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:45.116484 6463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:45.116575 6463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:45.173233 6456 heartbeater.cc:503] Master 127.9.225.62:42219 was elected leader, sending a full tablet report...
I20260809 00:57:45.190843 6463 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:45639
I20260809 00:57:45.190980 6668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:45639 every 8 connection(s)
I20260809 00:57:45.194025 6463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:57:45.196516 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 6463
I20260809 00:57:45.196832 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:57:45.197985 10116 external_mini_cluster.cc:1428] Setting key eedfd359f44f6f52df7aa9bc4656cf38
I20260809 00:57:45.208864 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42219
--builtin_ntp_servers=127.9.225.20:39247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 00:57:45.225284 6671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:45.197989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:52459 (local address 127.9.225.62:42219)
0809 00:57:45.202385 (+ 4396us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:45.202425 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:45.202497 (+ 72us) server_negotiation.cc:407] Connection header received
0809 00:57:45.207535 (+ 5038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:45.207569 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:45.207861 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:45.208057 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:45.213005 (+ 4948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:45.214984 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:45.219362 (+ 4378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:45.219861 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:45.220006 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:45.223174 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:45.223190 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:45.223196 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:45.223242 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:45.223304 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:45.223321 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:45.223415 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:45.224525 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0809 00:57:45.224706 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4181,"thread_start_us":3625,"threads_started":1}
I20260809 00:57:45.229482 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:45.198738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:42219 (local address 127.9.225.2:52459)
0809 00:57:45.200726 (+ 1988us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:45.200873 (+ 147us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:45.206671 (+ 5798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:45.208720 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:45.208763 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:45.209134 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:45.212502 (+ 3368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:45.212575 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:45.215369 (+ 2794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:45.215397 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:45.219025 (+ 3628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:45.219057 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:45.220180 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:45.220191 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:45.220578 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:45.222233 (+ 1655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:57:45.222252 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:45.222365 (+ 113us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:57:45.222622 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:45.223455 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:45.223482 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:45.223498 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 00:57:45.224272 (+ 774us) client_negotiation.cc:240] Negotiation successful
0809 00:57:45.225264 (+ 992us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":530,"threads_started":1}
I20260809 00:57:45.233054 6669 heartbeater.cc:347] Connected to a master server at 127.9.225.62:42219
I20260809 00:57:45.233834 6669 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:45.236306 6669 heartbeater.cc:511] Master 127.9.225.62:42219 requested a full tablet report, sending...
I20260809 00:57:45.239578 6146 ts_manager.cc:194] Registered new tserver with Master: 06dbdbc4c0f14796a91bea00ccf354c6 (127.9.225.2:45639)
I20260809 00:57:45.241241 6146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.225.2:52459
W20260809 00:57:45.928728 6673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:45.929163 6673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:45.929416 6673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:45.938211 6673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:45.938529 6673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:45.938747 6673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:57:45.949718 6673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:45.951349 6673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:45.953591 6673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:45.969288 6682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:45.969578 6681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:45.969976 6684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:45.970722 6673 server_base.cc:1034] running on GCE node
I20260809 00:57:45.971510 6673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:45.972731 6673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:45.973995 6673 hybrid_clock.cc:648] HybridClock initialized: now 1786237065973950 us; error 45 us; skew 500 ppm
I20260809 00:57:45.974402 6673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:57:45.977118 6673 webserver.cc:460] Webserver started at http://127.9.225.3:37467/ using document root <none> and password file <none>
I20260809 00:57:45.978039 6673 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:45.978181 6673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:45.978591 6673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:45.982461 6673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "8dd087de486142b39785990f5c9b2f0a"
format_stamp: "Formatted at 2026-08-09 00:57:45 on dist-test-slave-j2vl"
server_key: "74134ae9b4daf9e332fd1f5c969046db"
server_key_iv: "49177adf9f78feeb9cdb56c890406f4c"
server_key_version: "encryptionkey@0"
I20260809 00:57:45.983503 6673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "8dd087de486142b39785990f5c9b2f0a"
format_stamp: "Formatted at 2026-08-09 00:57:45 on dist-test-slave-j2vl"
server_key: "74134ae9b4daf9e332fd1f5c969046db"
server_key_iv: "49177adf9f78feeb9cdb56c890406f4c"
server_key_version: "encryptionkey@0"
I20260809 00:57:45.990800 6673 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260809 00:57:45.995867 6695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:45.997897 6673 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 00:57:45.998143 6673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "8dd087de486142b39785990f5c9b2f0a"
format_stamp: "Formatted at 2026-08-09 00:57:45 on dist-test-slave-j2vl"
server_key: "74134ae9b4daf9e332fd1f5c969046db"
server_key_iv: "49177adf9f78feeb9cdb56c890406f4c"
server_key_version: "encryptionkey@0"
I20260809 00:57:45.998420 6673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:46.018452 6673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:46.019626 6673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:46.020011 6673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:46.021394 6673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:46.023311 6673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:46.023486 6673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:46.023653 6673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:46.023785 6673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:46.067199 6673 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:43599
I20260809 00:57:46.067263 6855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:43599 every 8 connection(s)
I20260809 00:57:46.068864 6673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:57:46.074929 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 6673
I20260809 00:57:46.075150 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:57:46.075699 10116 external_mini_cluster.cc:1428] Setting key 5e3960c39ef0d3c918d73576bcba6cf1
I20260809 00:57:46.085009 6858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.072270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:60501 (local address 127.9.225.62:42219)
0809 00:57:46.072842 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:46.072850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:46.074492 (+ 1642us) server_negotiation.cc:407] Connection header received
0809 00:57:46.076688 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:46.076694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:46.076803 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:46.076957 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:46.078740 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.079842 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.081758 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.082212 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.082352 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.084033 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:46.084040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:46.084043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:46.084068 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:46.084116 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:46.084130 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:46.084206 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:46.084534 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 00:57:46.084650 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":275,"threads_started":1}
I20260809 00:57:46.086280 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.072996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:42219 (local address 127.9.225.3:60501)
0809 00:57:46.074239 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:46.074298 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:46.076244 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:46.077421 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:46.077438 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:46.077595 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:46.078503 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:46.078529 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.080056 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.080062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:46.081576 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:46.081592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.082489 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.082496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:46.082746 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.083601 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:57:46.083604 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:46.083655 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:57:46.083768 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:46.084231 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:46.084241 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:46.084249 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 00:57:46.084424 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 00:57:46.084814 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":428,"threads_started":1}
I20260809 00:57:46.088645 6856 heartbeater.cc:347] Connected to a master server at 127.9.225.62:42219
I20260809 00:57:46.088999 6856 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:46.089988 6856 heartbeater.cc:511] Master 127.9.225.62:42219 requested a full tablet report, sending...
I20260809 00:57:46.091984 6143 ts_manager.cc:194] Registered new tserver with Master: 8dd087de486142b39785990f5c9b2f0a (127.9.225.3:43599)
I20260809 00:57:46.093113 6143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.225.3:60501
I20260809 00:57:46.093175 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:57:46.107388 6858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.099580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.9.225.62:42219)
0809 00:57:46.099788 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:46.099793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:46.100566 (+ 773us) server_negotiation.cc:407] Connection header received
0809 00:57:46.100729 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:46.100733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:46.100810 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:46.100914 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:46.101698 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.102620 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.104575 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.105030 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.105153 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.106627 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:46.106632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:46.106636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:46.106659 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:46.106698 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:46.106711 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:46.106823 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:46.107027 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 00:57:46.107126 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 00:57:46.113196 6143 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:57:46.121733 6143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:57:46.153471 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.138652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:45639 (local address 127.0.0.1:57184)
0809 00:57:46.141106 (+ 2454us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:46.141177 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:46.142006 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:46.143933 (+ 1927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:46.143943 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:46.143982 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:46.144424 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:46.144439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.146643 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.146650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:46.149703 (+ 3053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:46.149721 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.152223 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.152230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:46.152487 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.152548 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 00:57:46.152697 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 00:57:46.152992 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":2,"thread_start_us":827,"threads_started":1}
I20260809 00:57:46.153986 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.138921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.9.225.2:45639)
0809 00:57:46.140856 (+ 1935us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:46.140868 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:46.142152 (+ 1284us) server_negotiation.cc:407] Connection header received
0809 00:57:46.142280 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:46.142289 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:46.143107 (+ 818us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:46.143330 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:46.144675 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.146398 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.149967 (+ 3569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.151568 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.151738 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.152036 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:46.153307 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0809 00:57:46.153584 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1741,"thread_start_us":1598,"threads_started":1}
I20260809 00:57:46.158166 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.139637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:43599 (local address 127.0.0.1:48026)
0809 00:57:46.141407 (+ 1770us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:46.141430 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:46.141699 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:46.150263 (+ 8564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:46.150268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:46.150298 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:46.150793 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:46.150808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.152625 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.152633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:46.154927 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:46.154939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.156936 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.156946 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:46.157136 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.157172 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 00:57:46.157449 (+ 277us) client_negotiation.cc:240] Negotiation successful
0809 00:57:46.157664 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1608,"mutex_wait_us":135,"thread_start_us":1356,"threads_started":1}
I20260809 00:57:46.158866 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.141248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.9.225.1:39817)
0809 00:57:46.147029 (+ 5781us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:46.147041 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:46.147931 (+ 890us) server_negotiation.cc:407] Connection header received
0809 00:57:46.148070 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:46.148088 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:46.148443 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:46.148687 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:46.149698 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.151448 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.154701 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.156322 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.156508 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.156893 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:46.158198 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0809 00:57:46.158418 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5496,"thread_start_us":5324,"threads_started":1}
I20260809 00:57:46.159312 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.140489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:39817 (local address 127.0.0.1:38442)
0809 00:57:46.147569 (+ 7080us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:46.147598 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:46.147769 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:46.148738 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:46.148747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:46.148792 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:46.149464 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:46.149484 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.151616 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.151623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:46.154437 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:46.154461 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.157016 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.157021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:46.157557 (+ 536us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.157589 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 00:57:46.157664 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 00:57:46.157802 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6913,"mutex_wait_us":34,"thread_start_us":6732,"threads_started":1}
I20260809 00:57:46.159732 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.140168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.9.225.3:43599)
0809 00:57:46.149628 (+ 9460us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:46.149638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:46.149676 (+ 38us) server_negotiation.cc:407] Connection header received
0809 00:57:46.149785 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:46.149794 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:46.150052 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:46.150283 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:46.151013 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.152448 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.155136 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.156679 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.156820 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.157109 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:46.158496 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0809 00:57:46.158696 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9255,"thread_start_us":9035,"threads_started":1}
I20260809 00:57:46.159281 6567 tablet_service.cc:1467] Processing CreateTablet for tablet e43df9d939424302822f4ee8f9b152d4 (DEFAULT_TABLE table=test-table [id=7d5c9b74dec54b57b68051cbf8cfc376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:57:46.161477 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43df9d939424302822f4ee8f9b152d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:46.164446 6762 tablet_service.cc:1467] Processing CreateTablet for tablet e43df9d939424302822f4ee8f9b152d4 (DEFAULT_TABLE table=test-table [id=7d5c9b74dec54b57b68051cbf8cfc376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:57:46.166373 6762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43df9d939424302822f4ee8f9b152d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:46.166234 6344 tablet_service.cc:1467] Processing CreateTablet for tablet e43df9d939424302822f4ee8f9b152d4 (DEFAULT_TABLE table=test-table [id=7d5c9b74dec54b57b68051cbf8cfc376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:57:46.168166 6344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e43df9d939424302822f4ee8f9b152d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:46.184453 6893 tablet_bootstrap.cc:492] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a: Bootstrap starting.
I20260809 00:57:46.186371 6894 tablet_bootstrap.cc:492] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6: Bootstrap starting.
I20260809 00:57:46.189200 6893 tablet_bootstrap.cc:654] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:46.191222 6893 log.cc:824] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:46.191602 6894 tablet_bootstrap.cc:654] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:46.194664 6894 log.cc:824] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:46.202483 6895 tablet_bootstrap.cc:492] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a: Bootstrap starting.
I20260809 00:57:46.209532 6895 tablet_bootstrap.cc:654] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:46.211378 6895 log.cc:824] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:46.212039 6893 tablet_bootstrap.cc:492] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a: No bootstrap required, opened a new log
I20260809 00:57:46.212040 6894 tablet_bootstrap.cc:492] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6: No bootstrap required, opened a new log
I20260809 00:57:46.212422 6893 ts_tablet_manager.cc:1397] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.014s
I20260809 00:57:46.212447 6894 ts_tablet_manager.cc:1397] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.009s
I20260809 00:57:46.216876 6895 tablet_bootstrap.cc:492] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a: No bootstrap required, opened a new log
I20260809 00:57:46.217258 6895 ts_tablet_manager.cc:1397] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260809 00:57:46.220912 6893 raft_consensus.cc:348] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 45639 } } peers { permanent_uuid: "883311029c2e48e5bab5a19f13ecb91a" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 39817 } } peers { permanent_uuid: "8dd087de486142b39785990f5c9b2f0a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43599 } }
I20260809 00:57:46.221443 6893 raft_consensus.cc:374] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:46.221560 6893 raft_consensus.cc:729] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd087de486142b39785990f5c9b2f0a, State: Initialized, Role: FOLLOWER
I20260809 00:57:46.222417 6893 consensus_queue.cc:260] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 45639 } } peers { permanent_uuid: "883311029c2e48e5bab5a19f13ecb91a" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 39817 } } peers { permanent_uuid: "8dd087de486142b39785990f5c9b2f0a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43599 } }
I20260809 00:57:46.223817 6856 heartbeater.cc:503] Master 127.9.225.62:42219 was elected leader, sending a full tablet report...
I20260809 00:57:46.224673 6893 ts_tablet_manager.cc:1428] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260809 00:57:46.225708 6894 raft_consensus.cc:348] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 45639 } } peers { permanent_uuid: "883311029c2e48e5bab5a19f13ecb91a" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 39817 } } peers { permanent_uuid: "8dd087de486142b39785990f5c9b2f0a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43599 } }
I20260809 00:57:46.226238 6894 raft_consensus.cc:374] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:46.226351 6894 raft_consensus.cc:729] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06dbdbc4c0f14796a91bea00ccf354c6, State: Initialized, Role: FOLLOWER
I20260809 00:57:46.227087 6894 consensus_queue.cc:260] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 45639 } } peers { permanent_uuid: "883311029c2e48e5bab5a19f13ecb91a" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 39817 } } peers { permanent_uuid: "8dd087de486142b39785990f5c9b2f0a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43599 } }
I20260809 00:57:46.229616 6895 raft_consensus.cc:348] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 45639 } } peers { permanent_uuid: "883311029c2e48e5bab5a19f13ecb91a" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 39817 } } peers { permanent_uuid: "8dd087de486142b39785990f5c9b2f0a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43599 } }
I20260809 00:57:46.230334 6895 raft_consensus.cc:374] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:46.230468 6895 raft_consensus.cc:729] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 883311029c2e48e5bab5a19f13ecb91a, State: Initialized, Role: FOLLOWER
I20260809 00:57:46.231364 6895 consensus_queue.cc:260] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 45639 } } peers { permanent_uuid: "883311029c2e48e5bab5a19f13ecb91a" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 39817 } } peers { permanent_uuid: "8dd087de486142b39785990f5c9b2f0a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43599 } }
I20260809 00:57:46.233980 6895 ts_tablet_manager.cc:1428] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260809 00:57:46.234432 6669 heartbeater.cc:503] Master 127.9.225.62:42219 was elected leader, sending a full tablet report...
I20260809 00:57:46.235193 6894 ts_tablet_manager.cc:1428] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6: Time spent starting tablet: real 0.023s user 0.016s sys 0.006s
W20260809 00:57:46.321116 6857 tablet.cc:2369] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:57:46.397718 6457 tablet.cc:2369] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:57:46.433075 6904 raft_consensus.cc:482] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:57:46.433481 6904 raft_consensus.cc:504] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 45639 } } peers { permanent_uuid: "883311029c2e48e5bab5a19f13ecb91a" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 39817 } } peers { permanent_uuid: "8dd087de486142b39785990f5c9b2f0a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43599 } }
I20260809 00:57:46.435460 6904 leader_election.cc:288] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 883311029c2e48e5bab5a19f13ecb91a (127.9.225.1:39817), 8dd087de486142b39785990f5c9b2f0a (127.9.225.3:43599)
I20260809 00:57:46.445016 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.436159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:42615 (local address 127.9.225.1:39817)
0809 00:57:46.436506 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:46.436515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:46.437203 (+ 688us) server_negotiation.cc:407] Connection header received
0809 00:57:46.437301 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:46.437307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:46.437425 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:46.437536 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:46.438441 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.439634 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.442987 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.443889 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.444056 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.444175 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:46.444622 (+ 447us) server_negotiation.cc:299] Negotiation successful
0809 00:57:46.444772 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260809 00:57:46.445165 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.436114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:58725 (local address 127.9.225.3:43599)
0809 00:57:46.436369 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:46.436377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:46.437068 (+ 691us) server_negotiation.cc:407] Connection header received
0809 00:57:46.437302 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:46.437307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:46.437424 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:46.437536 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:46.438441 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.439634 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.442695 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.444022 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.444157 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.444259 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:46.444761 (+ 502us) server_negotiation.cc:299] Negotiation successful
0809 00:57:46.444911 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260809 00:57:46.445163 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.435970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:39817 (local address 127.9.225.2:42615)
0809 00:57:46.437051 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:46.437067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:46.437173 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:46.437569 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:46.437574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:46.437613 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:46.438212 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:46.438231 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.439789 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.439795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:46.442763 (+ 2968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:46.442781 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.444120 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.444126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:46.444387 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.444426 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 00:57:46.444535 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 00:57:46.444703 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":211,"spinlock_wait_cycles":896,"thread_start_us":563,"threads_started":1}
I20260809 00:57:46.445870 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.435839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:43599 (local address 127.9.225.2:58725)
0809 00:57:46.436896 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:46.436919 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:46.437117 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:46.437569 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:46.437574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:46.437619 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:46.438238 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:46.438247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.439793 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.439798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:46.442346 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:46.442370 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.444321 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.444328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:46.444559 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.444589 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 00:57:46.445388 (+ 799us) client_negotiation.cc:240] Negotiation successful
0809 00:57:46.445546 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":245,"spinlock_wait_cycles":9728,"thread_start_us":768,"threads_started":1}
I20260809 00:57:46.446494 6386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43df9d939424302822f4ee8f9b152d4" candidate_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "883311029c2e48e5bab5a19f13ecb91a" is_pre_election: true
I20260809 00:57:46.447101 6386 raft_consensus.cc:2454] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06dbdbc4c0f14796a91bea00ccf354c6 in term 0.
W20260809 00:57:46.447628 6670 tablet.cc:2369] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:57:46.448308 6489 leader_election.cc:302] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06dbdbc4c0f14796a91bea00ccf354c6, 883311029c2e48e5bab5a19f13ecb91a; no voters:
I20260809 00:57:46.448372 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43df9d939424302822f4ee8f9b152d4" candidate_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd087de486142b39785990f5c9b2f0a" is_pre_election: true
I20260809 00:57:46.448840 6904 raft_consensus.cc:2781] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:57:46.448998 6904 raft_consensus.cc:482] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:57:46.449011 6790 raft_consensus.cc:2454] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06dbdbc4c0f14796a91bea00ccf354c6 in term 0.
I20260809 00:57:46.449126 6904 raft_consensus.cc:3037] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:46.451566 6904 raft_consensus.cc:504] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 45639 } } peers { permanent_uuid: "883311029c2e48e5bab5a19f13ecb91a" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 39817 } } peers { permanent_uuid: "8dd087de486142b39785990f5c9b2f0a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43599 } }
I20260809 00:57:46.452450 6904 leader_election.cc:288] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [CANDIDATE]: Term 1 election: Requested vote from peers 883311029c2e48e5bab5a19f13ecb91a (127.9.225.1:39817), 8dd087de486142b39785990f5c9b2f0a (127.9.225.3:43599)
I20260809 00:57:46.453080 6790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43df9d939424302822f4ee8f9b152d4" candidate_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd087de486142b39785990f5c9b2f0a"
I20260809 00:57:46.453090 6386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e43df9d939424302822f4ee8f9b152d4" candidate_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "883311029c2e48e5bab5a19f13ecb91a"
I20260809 00:57:46.453313 6386 raft_consensus.cc:3037] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:46.453320 6790 raft_consensus.cc:3037] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:46.455955 6790 raft_consensus.cc:2454] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06dbdbc4c0f14796a91bea00ccf354c6 in term 1.
I20260809 00:57:46.456604 6491 leader_election.cc:302] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06dbdbc4c0f14796a91bea00ccf354c6, 8dd087de486142b39785990f5c9b2f0a; no voters:
I20260809 00:57:46.456830 6386 raft_consensus.cc:2454] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06dbdbc4c0f14796a91bea00ccf354c6 in term 1.
I20260809 00:57:46.457020 6904 raft_consensus.cc:2781] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:57:46.457818 6904 raft_consensus.cc:686] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 1 LEADER]: Becoming Leader. State: Replica: 06dbdbc4c0f14796a91bea00ccf354c6, State: Running, Role: LEADER
I20260809 00:57:46.458405 6904 consensus_queue.cc:237] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 45639 } } peers { permanent_uuid: "883311029c2e48e5bab5a19f13ecb91a" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 39817 } } peers { permanent_uuid: "8dd087de486142b39785990f5c9b2f0a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43599 } }
I20260809 00:57:46.464773 6143 catalog_manager.cc:5495] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06dbdbc4c0f14796a91bea00ccf354c6 (127.9.225.2). New cstate: current_term: 1 leader_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dbdbc4c0f14796a91bea00ccf354c6" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 45639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "883311029c2e48e5bab5a19f13ecb91a" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dd087de486142b39785990f5c9b2f0a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43599 } health_report { overall_health: UNKNOWN } } }
I20260809 00:57:46.531348 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.524812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.9.225.2:45639)
0809 00:57:46.525116 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:46.525123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:46.525146 (+ 23us) server_negotiation.cc:407] Connection header received
0809 00:57:46.525460 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:46.525465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:46.525578 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:46.525760 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:57:46.526182 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.526934 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.528602 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.529046 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.529196 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.529919 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:57:46.530626 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:57:46.530820 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:46.530937 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 00:57:46.531035 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260809 00:57:46.541508 6790 raft_consensus.cc:1261] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a [term 1 FOLLOWER]: Refusing update from remote peer 06dbdbc4c0f14796a91bea00ccf354c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:57:46.541522 6386 raft_consensus.cc:1261] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a [term 1 FOLLOWER]: Refusing update from remote peer 06dbdbc4c0f14796a91bea00ccf354c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:57:46.542690 6912 consensus_queue.cc:1035] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dd087de486142b39785990f5c9b2f0a" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 43599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:57:46.543123 6904 consensus_queue.cc:1035] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "883311029c2e48e5bab5a19f13ecb91a" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:57:46.556841 6915 mvcc.cc:205] Tried to move back new op lower bound from 7316427024540626944 to 7316427024218050560. Current Snapshot: MvccSnapshot[applied={T|T < 7316427024540626944}]
I20260809 00:57:46.573148 6146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:57:46.573558 6146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:57:46.577971 6146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8cbd2af3cf444afd8053ae62e5cc5e3b: Sending DeleteTablet for 3 replicas of tablet e43df9d939424302822f4ee8f9b152d4
I20260809 00:57:46.579605 6762 tablet_service.cc:1514] Processing DeleteTablet for tablet e43df9d939424302822f4ee8f9b152d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:57:46 UTC) from {username='slave'} at 127.0.0.1:48026
I20260809 00:57:46.579595 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet e43df9d939424302822f4ee8f9b152d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:57:46 UTC) from {username='slave'} at 127.0.0.1:57184
I20260809 00:57:46.579670 6344 tablet_service.cc:1514] Processing DeleteTablet for tablet e43df9d939424302822f4ee8f9b152d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:57:46 UTC) from {username='slave'} at 127.0.0.1:38442
I20260809 00:57:46.583139 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 6234
I20260809 00:57:46.595340 6932 tablet_replica.cc:330] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a: stopping tablet replica
I20260809 00:57:46.595995 6932 raft_consensus.cc:2229] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:57:46.596655 6932 raft_consensus.cc:2258] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:57:46.598642 6931 tablet_replica.cc:330] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a: stopping tablet replica
I20260809 00:57:46.599221 6931 raft_consensus.cc:2229] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:57:46.599880 6931 raft_consensus.cc:2258] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:57:46.600989 6930 tablet_replica.cc:330] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6: stopping tablet replica
I20260809 00:57:46.601999 6930 raft_consensus.cc:2229] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:57:46.602828 6930 raft_consensus.cc:2258] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:57:46.603045 6931 ts_tablet_manager.cc:1905] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:57:46.603081 6932 ts_tablet_manager.cc:1905] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:57:46.605427 6930 ts_tablet_manager.cc:1905] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:57:46.610901 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.585317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.9.225.1:39817)
0809 00:57:46.585602 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:46.585612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:46.597839 (+ 12227us) server_negotiation.cc:407] Connection header received
0809 00:57:46.598096 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:46.598104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:46.598282 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:46.598427 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:46.600175 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.601480 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.604105 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.604850 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.604974 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.608165 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:46.608184 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:46.608206 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:46.609185 (+ 979us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:46.609282 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:46.609320 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:46.609460 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:46.609792 (+ 332us) server_negotiation.cc:299] Negotiation successful
0809 00:57:46.609956 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260809 00:57:46.611320 6931 ts_tablet_manager.cc:1918] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:57:46.611531 6931 log.cc:1196] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/e43df9d939424302822f4ee8f9b152d4
I20260809 00:57:46.611877 6932 ts_tablet_manager.cc:1918] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:57:46.612078 6932 log.cc:1196] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/e43df9d939424302822f4ee8f9b152d4
I20260809 00:57:46.612120 6931 ts_tablet_manager.cc:1939] T e43df9d939424302822f4ee8f9b152d4 P 8dd087de486142b39785990f5c9b2f0a: Deleting consensus metadata
I20260809 00:57:46.612702 6932 ts_tablet_manager.cc:1939] T e43df9d939424302822f4ee8f9b152d4 P 883311029c2e48e5bab5a19f13ecb91a: Deleting consensus metadata
I20260809 00:57:46.613175 6450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:46.613448 6930 ts_tablet_manager.cc:1918] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:57:46.613646 6930 log.cc:1196] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/e43df9d939424302822f4ee8f9b152d4
I20260809 00:57:46.614225 6930 ts_tablet_manager.cc:1939] T e43df9d939424302822f4ee8f9b152d4 P 06dbdbc4c0f14796a91bea00ccf354c6: Deleting consensus metadata
I20260809 00:57:46.614492 6120 catalog_manager.cc:4841] TS 8dd087de486142b39785990f5c9b2f0a (127.9.225.3:43599): tablet e43df9d939424302822f4ee8f9b152d4 (table test-table [id=7d5c9b74dec54b57b68051cbf8cfc376]) successfully deleted
I20260809 00:57:46.616393 6119 catalog_manager.cc:4841] TS 06dbdbc4c0f14796a91bea00ccf354c6 (127.9.225.2:45639): tablet e43df9d939424302822f4ee8f9b152d4 (table test-table [id=7d5c9b74dec54b57b68051cbf8cfc376]) successfully deleted
I20260809 00:57:46.718880 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 6234
I20260809 00:57:46.719091 6119 catalog_manager.cc:4841] TS 883311029c2e48e5bab5a19f13ecb91a (127.9.225.1:39817): tablet e43df9d939424302822f4ee8f9b152d4 (table test-table [id=7d5c9b74dec54b57b68051cbf8cfc376]) successfully deleted
I20260809 00:57:46.756400 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 6463
I20260809 00:57:46.765305 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.757404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.9.225.2:45639)
0809 00:57:46.757629 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:46.757635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:46.757659 (+ 24us) server_negotiation.cc:407] Connection header received
0809 00:57:46.757805 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:46.757811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:46.757934 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:46.758064 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:46.758480 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.759764 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.761226 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.761902 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.762048 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.763281 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:46.763294 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:46.763323 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:46.764268 (+ 945us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:46.764344 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:46.764374 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:46.764501 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:46.764748 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 00:57:46.764895 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 00:57:46.766223 6656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:46.890918 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 6463
I20260809 00:57:46.922945 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 6673
I20260809 00:57:46.930588 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:46.923776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.9.225.3:43599)
0809 00:57:46.923963 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:46.923967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:46.924023 (+ 56us) server_negotiation.cc:407] Connection header received
0809 00:57:46.924180 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:46.924184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:46.924288 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:46.924380 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:46.924836 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.925718 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.927206 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:46.927668 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:46.927782 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:46.928955 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:46.928972 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:46.928994 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:46.929856 (+ 862us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:46.929910 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:46.929933 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:46.930030 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:46.930203 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 00:57:46.930320 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 00:57:46.931380 6850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:47.026350 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 6673
I20260809 00:57:47.052722 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 6091
I20260809 00:57:47.054132 6188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:47.188169 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 6091
2026-08-09T00:57:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (4359 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:57:47 dist-test-slave-j2vl krb5kdc[6949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:57:47 dist-test-slave-j2vl krb5kdc[6949](info): set up 2 sockets
Aug 09 00:57:47 dist-test-slave-j2vl krb5kdc[6949](info): commencing operation
krb5kdc: starting...
W20260809 00:57:48.613997 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.369s user 0.006s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:57:48 dist-test-slave-j2vl krb5kdc[6949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:57:48.990190 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43477
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:36169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:43477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:57:49.398123 6965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:49.398502 6965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:49.398674 6965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:49.407225 6965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:57:49.407418 6965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:49.407536 6965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:49.407648 6965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:57:49.407737 6965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:57:49.419180 6965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:43477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:49.420868 6965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:49.422416 6965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:49.437255 6971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:49.437321 6972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:49.437402 6974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:49.438086 6965 server_base.cc:1034] running on GCE node
I20260809 00:57:49.439385 6965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:49.441133 6965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:49.442531 6965 hybrid_clock.cc:648] HybridClock initialized: now 1786237069442425 us; error 57 us; skew 500 ppm
I20260809 00:57:49.443272 6965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:49 dist-test-slave-j2vl krb5kdc[6949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237069, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:49.447392 6965 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:57:49.449280 6965 webserver.cc:460] Webserver started at http://127.9.225.62:43635/ using document root <none> and password file <none>
I20260809 00:57:49.450176 6965 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:49.450353 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:49.450707 6965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:49.454301 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "7dbf18182fe34b5b8c5bf9a3e5640282"
format_stamp: "Formatted at 2026-08-09 00:57:49 on dist-test-slave-j2vl"
server_key: "4922d8ffd70c7130f4f18cd83117f21c"
server_key_iv: "9d238fd99fca0f34524fa47c03017c02"
server_key_version: "encryptionkey@0"
I20260809 00:57:49.455240 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "7dbf18182fe34b5b8c5bf9a3e5640282"
format_stamp: "Formatted at 2026-08-09 00:57:49 on dist-test-slave-j2vl"
server_key: "4922d8ffd70c7130f4f18cd83117f21c"
server_key_iv: "9d238fd99fca0f34524fa47c03017c02"
server_key_version: "encryptionkey@0"
I20260809 00:57:49.461844 6965 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 00:57:49.468642 6984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:49.473886 6965 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260809 00:57:49.474285 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "7dbf18182fe34b5b8c5bf9a3e5640282"
format_stamp: "Formatted at 2026-08-09 00:57:49 on dist-test-slave-j2vl"
server_key: "4922d8ffd70c7130f4f18cd83117f21c"
server_key_iv: "9d238fd99fca0f34524fa47c03017c02"
server_key_version: "encryptionkey@0"
I20260809 00:57:49.474709 6965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:49.511090 6965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:49.516592 6965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:49.516949 6965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:49.539016 6965 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:43477
I20260809 00:57:49.539057 7065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:43477 every 8 connection(s)
I20260809 00:57:49.540603 6965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:57:49.542382 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 6965
I20260809 00:57:49.542606 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:57:49.543315 10116 external_mini_cluster.cc:1428] Setting key 6308f2d5fd265b1adedba6f21b3dd836
I20260809 00:57:49.546416 7066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 00:57:49 dist-test-slave-j2vl krb5kdc[6949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:57:49.560294 7066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282: Bootstrap starting.
I20260809 00:57:49.564553 7066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:49.565896 7066 log.cc:824] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:49.570143 7066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282: No bootstrap required, opened a new log
I20260809 00:57:49.570799 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:49.545644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.9.225.62:43477)
0809 00:57:49.546852 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:49.546872 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:49.546927 (+ 55us) server_negotiation.cc:407] Connection header received
0809 00:57:49.548803 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:49.548831 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:49.549308 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:49.549940 (+ 632us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:49.551733 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:49.553192 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:49.554753 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:49.555506 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:49.555735 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:49.561510 (+ 5775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:49.561523 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:49.561542 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:49.561595 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:49.565153 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:49.566213 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:49.566225 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:49.566235 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:49.566505 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:49.567115 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:49.567119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:49.567121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:49.567897 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:49.568233 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:49.568724 (+ 491us) server_negotiation.cc:299] Negotiation successful
0809 00:57:49.569251 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":607,"threads_started":1}
I20260809 00:57:49.579353 7066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbf18182fe34b5b8c5bf9a3e5640282" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43477 } }
I20260809 00:57:49.579856 7066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:49.580000 7066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dbf18182fe34b5b8c5bf9a3e5640282, State: Initialized, Role: FOLLOWER
I20260809 00:57:49.580816 7066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbf18182fe34b5b8c5bf9a3e5640282" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43477 } }
I20260809 00:57:49.581142 7066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:57:49.581342 7066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:57:49.581624 7066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:49.584419 7066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbf18182fe34b5b8c5bf9a3e5640282" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43477 } }
I20260809 00:57:49.585059 7066 leader_election.cc:302] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dbf18182fe34b5b8c5bf9a3e5640282; no voters:
I20260809 00:57:49.585826 7066 leader_election.cc:288] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:57:49.586010 7072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:57:49.586856 7072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [term 1 LEADER]: Becoming Leader. State: Replica: 7dbf18182fe34b5b8c5bf9a3e5640282, State: Running, Role: LEADER
I20260809 00:57:49.587527 7072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbf18182fe34b5b8c5bf9a3e5640282" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43477 } }
I20260809 00:57:49.588325 7066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:57:49.591794 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dbf18182fe34b5b8c5bf9a3e5640282. Latest consensus state: current_term: 1 leader_uuid: "7dbf18182fe34b5b8c5bf9a3e5640282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbf18182fe34b5b8c5bf9a3e5640282" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43477 } } }
I20260809 00:57:49.592372 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [sys.catalog]: This master's current role is: LEADER
I20260809 00:57:49.591850 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dbf18182fe34b5b8c5bf9a3e5640282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbf18182fe34b5b8c5bf9a3e5640282" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43477 } } }
I20260809 00:57:49.592724 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282 [sys.catalog]: This master's current role is: LEADER
I20260809 00:57:49.594802 7090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:57:49.600622 7090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:57:49.611896 7090 catalog_manager.cc:1347] Generated new cluster ID: 2d3e1a6c3afe4391bb24b5c87cbd96bd
I20260809 00:57:49.612051 7090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:57:49.625810 7090 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:57:49.627300 7090 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:57:49.635630 7090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282: Generated new TSK 0
I20260809 00:57:49.636951 7090 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:57:49.730331 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43477
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:36169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:57:50.141105 7113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:50.141481 7113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:50.141588 7113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:50.149216 7113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:50.149391 7113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:50.149516 7113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:57:50.160076 7113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:50.161603 7113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:50.163070 7113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:50.178745 7123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:50.178654 7124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:50.178685 7128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:50.179690 7113 server_base.cc:1034] running on GCE node
I20260809 00:57:50.180485 7113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:50.181686 7113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:50.182950 7113 hybrid_clock.cc:648] HybridClock initialized: now 1786237070182903 us; error 58 us; skew 500 ppm
I20260809 00:57:50.183348 7113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:50 dist-test-slave-j2vl krb5kdc[6949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237070, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:50.187800 7113 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:57:50.190070 7113 webserver.cc:460] Webserver started at http://127.9.225.1:45855/ using document root <none> and password file <none>
I20260809 00:57:50.190953 7113 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:50.191131 7113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:50.191779 7113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:50.195712 7113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673"
format_stamp: "Formatted at 2026-08-09 00:57:50 on dist-test-slave-j2vl"
server_key: "9f2c97cf1036da7f7ee479e63892e6f2"
server_key_iv: "e5de2d025021e47d79b777c3e09a0c23"
server_key_version: "encryptionkey@0"
I20260809 00:57:50.196700 7113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673"
format_stamp: "Formatted at 2026-08-09 00:57:50 on dist-test-slave-j2vl"
server_key: "9f2c97cf1036da7f7ee479e63892e6f2"
server_key_iv: "e5de2d025021e47d79b777c3e09a0c23"
server_key_version: "encryptionkey@0"
I20260809 00:57:50.203135 7113 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 00:57:50.207544 7139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:50.209307 7113 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260809 00:57:50.209575 7113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673"
format_stamp: "Formatted at 2026-08-09 00:57:50 on dist-test-slave-j2vl"
server_key: "9f2c97cf1036da7f7ee479e63892e6f2"
server_key_iv: "e5de2d025021e47d79b777c3e09a0c23"
server_key_version: "encryptionkey@0"
I20260809 00:57:50.209857 7113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:50.232317 7113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:50.237861 7113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:50.238269 7113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:50.239464 7113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:50.241164 7113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:50.241250 7113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:50.241429 7113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:50.241500 7113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 00:57:50.276898 7113 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:37955
I20260809 00:57:50.276945 7326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:37955 every 8 connection(s)
I20260809 00:57:50.278544 7113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:57:50.279196 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 7113
I20260809 00:57:50.279448 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:57:50.280085 10116 external_mini_cluster.cc:1428] Setting key b506bde53a1cf05554ce53cc12b8ccd8
Aug 09 00:57:50 dist-test-slave-j2vl krb5kdc[6949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237070, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:57:50.307672 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:50.282095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:40349 (local address 127.9.225.62:43477)
0809 00:57:50.285785 (+ 3690us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:50.285790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:50.285812 (+ 22us) server_negotiation.cc:407] Connection header received
0809 00:57:50.288106 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:50.288111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:50.288194 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:50.288307 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:50.290897 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:50.291706 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:50.294609 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:50.295151 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:50.295286 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:50.300532 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:50.300538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:50.300542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:50.300583 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:50.303564 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:50.304792 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:50.304797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:50.304799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:50.304948 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:50.305582 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:50.305586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:50.305588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:50.305910 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:50.306059 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:50.307164 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0809 00:57:50.307375 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3560,"thread_start_us":3437,"threads_started":1}
I20260809 00:57:50.309409 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:50.283089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43477 (local address 127.9.225.1:40349)
0809 00:57:50.284656 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:50.284720 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:50.287657 (+ 2937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:50.288672 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:50.288687 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:50.289658 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:50.290626 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:50.290661 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:50.291927 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:50.291935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:50.294379 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:50.294402 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:50.295452 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:50.295463 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:50.295832 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:50.296718 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:57:50.296764 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:50.300164 (+ 3400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:50.303769 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:50.303784 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:50.303809 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:50.304601 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:50.305071 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:50.305079 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:50.305085 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:50.305413 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:50.306084 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:50.306097 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:50.306679 (+ 582us) client_negotiation.cc:769] Sending connection context
0809 00:57:50.307037 (+ 358us) client_negotiation.cc:240] Negotiation successful
0809 00:57:50.307699 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"thread_start_us":822,"threads_started":1}
I20260809 00:57:50.311471 7327 heartbeater.cc:347] Connected to a master server at 127.9.225.62:43477
I20260809 00:57:50.311877 7327 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:50.312749 7327 heartbeater.cc:511] Master 127.9.225.62:43477 requested a full tablet report, sending...
I20260809 00:57:50.315743 7014 ts_manager.cc:194] Registered new tserver with Master: ad6b0c93bd8848aaa1e3ce092cbfa673 (127.9.225.1:37955)
I20260809 00:57:50.318229 7014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:40349
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:57:50.366287 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43477
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:36169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:57:50.760187 7336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:50.760541 7336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:50.760710 7336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:50.768533 7336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:50.768684 7336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:50.768882 7336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:57:50.779953 7336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:50.781692 7336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:50.783344 7336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:50.798907 7348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:50.799005 7349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:50.799036 7352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:50.803090 7336 server_base.cc:1034] running on GCE node
I20260809 00:57:50.803825 7336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:50.804883 7336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:50.806125 7336 hybrid_clock.cc:648] HybridClock initialized: now 1786237070806077 us; error 32 us; skew 500 ppm
I20260809 00:57:50.806466 7336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:50 dist-test-slave-j2vl krb5kdc[6949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237070, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:50.810112 7336 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:57:50.812003 7336 webserver.cc:460] Webserver started at http://127.9.225.2:43109/ using document root <none> and password file <none>
I20260809 00:57:50.812961 7336 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:50.813139 7336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:50.813558 7336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:50.817173 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "fd3593e08a424e5c9771486c60ed3424"
format_stamp: "Formatted at 2026-08-09 00:57:50 on dist-test-slave-j2vl"
server_key: "81cd5464599b1ed3fb8a425b926e3a80"
server_key_iv: "5e5b4ed06da7914d5a98d2a015b4ca01"
server_key_version: "encryptionkey@0"
I20260809 00:57:50.818115 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "fd3593e08a424e5c9771486c60ed3424"
format_stamp: "Formatted at 2026-08-09 00:57:50 on dist-test-slave-j2vl"
server_key: "81cd5464599b1ed3fb8a425b926e3a80"
server_key_iv: "5e5b4ed06da7914d5a98d2a015b4ca01"
server_key_version: "encryptionkey@0"
I20260809 00:57:50.824213 7336 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:57:50.828353 7360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:50.829977 7336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 00:57:50.830202 7336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "fd3593e08a424e5c9771486c60ed3424"
format_stamp: "Formatted at 2026-08-09 00:57:50 on dist-test-slave-j2vl"
server_key: "81cd5464599b1ed3fb8a425b926e3a80"
server_key_iv: "5e5b4ed06da7914d5a98d2a015b4ca01"
server_key_version: "encryptionkey@0"
I20260809 00:57:50.830494 7336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:50.871074 7336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:50.876618 7336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:50.877027 7336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:50.878224 7336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:50.879894 7336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:50.880016 7336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:50.880187 7336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:50.880265 7336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:50.920939 7336 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:36245
I20260809 00:57:50.920979 7547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:36245 every 8 connection(s)
I20260809 00:57:50.922531 7336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:57:50.925774 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 7336
I20260809 00:57:50.926049 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:57:50.926718 10116 external_mini_cluster.cc:1428] Setting key abe77e4e73b134f9d1a06871b84410aa
Aug 09 00:57:50 dist-test-slave-j2vl krb5kdc[6949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237070, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:57:50.948287 7551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:50.925840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:53315 (local address 127.9.225.62:43477)
0809 00:57:50.926437 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:50.926446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:50.927655 (+ 1209us) server_negotiation.cc:407] Connection header received
0809 00:57:50.929783 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:50.929790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:50.929905 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:50.930050 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:50.931804 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:50.933179 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:50.935209 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:50.936180 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:50.936355 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:50.940839 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:50.940845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:50.940849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:50.940903 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:50.943856 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:50.945020 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:50.945028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:50.945033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:50.945216 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:50.945888 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:50.945893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:50.945896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:50.946336 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:50.946507 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:50.947424 (+ 917us) server_negotiation.cc:299] Negotiation successful
0809 00:57:50.947753 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":277,"threads_started":1}
I20260809 00:57:50.949267 7552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:50.926518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43477 (local address 127.9.225.2:53315)
0809 00:57:50.927453 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:50.927509 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:50.929465 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:50.930294 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:50.930304 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:50.930934 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:50.931587 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:50.931605 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:50.933458 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:50.933463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:50.934975 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:50.934991 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:50.936481 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:50.936487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:50.936725 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:50.937548 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:57:50.937573 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:50.940445 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:50.944118 (+ 3673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:50.944134 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:50.944163 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:50.944836 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:50.945467 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:50.945475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:50.945478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:50.945735 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:50.946507 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:50.946515 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:50.946925 (+ 410us) client_negotiation.cc:769] Sending connection context
0809 00:57:50.947255 (+ 330us) client_negotiation.cc:240] Negotiation successful
0809 00:57:50.947758 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":428,"threads_started":1}
I20260809 00:57:50.951335 7548 heartbeater.cc:347] Connected to a master server at 127.9.225.62:43477
I20260809 00:57:50.951782 7548 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:50.952915 7548 heartbeater.cc:511] Master 127.9.225.62:43477 requested a full tablet report, sending...
I20260809 00:57:50.955083 7014 ts_manager.cc:194] Registered new tserver with Master: fd3593e08a424e5c9771486c60ed3424 (127.9.225.2:36245)
I20260809 00:57:50.956046 7014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:53315
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:57:51.013121 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43477
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:36169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:57:51.322268 7327 heartbeater.cc:503] Master 127.9.225.62:43477 was elected leader, sending a full tablet report...
W20260809 00:57:51.420861 7558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:51.421182 7558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:51.421284 7558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:51.429184 7558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:51.429392 7558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:51.429538 7558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:57:51.440317 7558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:36169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:43477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:51.441850 7558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:51.443267 7558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:51.458671 7565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:51.458622 7569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:51.458732 7564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:51.460065 7558 server_base.cc:1034] running on GCE node
I20260809 00:57:51.460845 7558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:51.462103 7558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:51.463339 7558 hybrid_clock.cc:648] HybridClock initialized: now 1786237071463261 us; error 69 us; skew 500 ppm
I20260809 00:57:51.463690 7558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:51 dist-test-slave-j2vl krb5kdc[6949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237071, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:51.470763 7558 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:57:51.472617 7558 webserver.cc:460] Webserver started at http://127.9.225.3:38879/ using document root <none> and password file <none>
I20260809 00:57:51.473477 7558 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:51.473614 7558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:51.473942 7558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:51.477156 7558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "ad27bb74ac09448daac545370e1f60ff"
format_stamp: "Formatted at 2026-08-09 00:57:51 on dist-test-slave-j2vl"
server_key: "318b8f0341b28229f6ddd92b23298f9a"
server_key_iv: "ecd11e60ec2fa3b6f701f677abfeb4e0"
server_key_version: "encryptionkey@0"
I20260809 00:57:51.478117 7558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "ad27bb74ac09448daac545370e1f60ff"
format_stamp: "Formatted at 2026-08-09 00:57:51 on dist-test-slave-j2vl"
server_key: "318b8f0341b28229f6ddd92b23298f9a"
server_key_iv: "ecd11e60ec2fa3b6f701f677abfeb4e0"
server_key_version: "encryptionkey@0"
I20260809 00:57:51.484553 7558 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 00:57:51.488782 7576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:51.490664 7558 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:57:51.490864 7558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "ad27bb74ac09448daac545370e1f60ff"
format_stamp: "Formatted at 2026-08-09 00:57:51 on dist-test-slave-j2vl"
server_key: "318b8f0341b28229f6ddd92b23298f9a"
server_key_iv: "ecd11e60ec2fa3b6f701f677abfeb4e0"
server_key_version: "encryptionkey@0"
I20260809 00:57:51.491078 7558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:51.507318 7558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:51.512488 7558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:51.512781 7558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:51.514127 7558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:51.515784 7558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:51.515872 7558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:51.515967 7558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:51.516016 7558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:51.554574 7558 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:34623
I20260809 00:57:51.554622 7764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:34623 every 8 connection(s)
I20260809 00:57:51.556150 7558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:57:51.561779 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 7558
I20260809 00:57:51.562009 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:57:51.562508 10116 external_mini_cluster.cc:1428] Setting key 1ba1a5296b98a803dcf7f3010903a5b0
Aug 09 00:57:51 dist-test-slave-j2vl krb5kdc[6949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237071, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:57:51.578622 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.559273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:42937 (local address 127.9.225.62:43477)
0809 00:57:51.559799 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:51.559806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:51.560929 (+ 1123us) server_negotiation.cc:407] Connection header received
0809 00:57:51.563090 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:51.563096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:51.563200 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:51.563335 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:51.565102 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.566135 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.567840 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.568253 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.568393 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.572245 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:51.572250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:51.572254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:51.572294 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:51.575101 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:51.575941 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:51.575948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:51.575951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:51.576088 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:51.576567 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:51.576572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:51.576576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:51.576913 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:51.577086 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:51.577957 (+ 871us) server_negotiation.cc:299] Negotiation successful
0809 00:57:51.578209 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":250,"threads_started":1}
I20260809 00:57:51.579772 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.559916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:43477 (local address 127.9.225.3:42937)
0809 00:57:51.560737 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:51.560786 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:51.562748 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:51.563632 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:51.563642 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:51.564237 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:51.564907 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:51.564928 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.566322 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.566327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:51.567660 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:51.567675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.568506 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.568516 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:51.568783 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.569582 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:57:51.569608 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:51.571972 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:51.575280 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:51.575292 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:51.575328 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:51.575808 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:51.576207 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:51.576212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:51.576216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:51.576441 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:51.577051 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:51.577060 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:51.577534 (+ 474us) client_negotiation.cc:769] Sending connection context
0809 00:57:51.577835 (+ 301us) client_negotiation.cc:240] Negotiation successful
0809 00:57:51.578474 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":324,"threads_started":1}
I20260809 00:57:51.581501 7765 heartbeater.cc:347] Connected to a master server at 127.9.225.62:43477
I20260809 00:57:51.581894 7765 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:51.582796 7765 heartbeater.cc:511] Master 127.9.225.62:43477 requested a full tablet report, sending...
I20260809 00:57:51.584569 7014 ts_manager.cc:194] Registered new tserver with Master: ad27bb74ac09448daac545370e1f60ff (127.9.225.3:34623)
I20260809 00:57:51.585494 7014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:42937
I20260809 00:57:51.591245 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:57:51.613845 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.596563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.9.225.62:43477)
0809 00:57:51.596830 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:51.596841 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:51.597380 (+ 539us) server_negotiation.cc:407] Connection header received
0809 00:57:51.597642 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:51.597647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:51.597739 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:51.597840 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:51.599670 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.600981 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.602661 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.603239 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.603416 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.607094 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:51.607101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:51.607105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:51.607150 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:51.610206 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:51.611279 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:51.611284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:51.611287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:51.611436 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:51.612002 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:51.612007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:51.612009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:51.612376 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:51.612710 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:51.613205 (+ 495us) server_negotiation.cc:299] Negotiation successful
0809 00:57:51.613486 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 00:57:51.618571 7014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:57:51.626538 7014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:57:51.655000 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.642335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:34623 (local address 127.0.0.1:39790)
0809 00:57:51.645997 (+ 3662us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:51.646018 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:51.646251 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:51.647286 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:51.647291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:51.647317 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:51.647827 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:51.647845 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.649622 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.649630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:51.651818 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:51.651835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.653811 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.653818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:51.654055 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.654127 (+ 72us) client_negotiation.cc:769] Sending connection context
0809 00:57:51.654312 (+ 185us) client_negotiation.cc:240] Negotiation successful
0809 00:57:51.654558 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3530,"mutex_wait_us":39,"thread_start_us":992,"threads_started":1}
I20260809 00:57:51.654968 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.642997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.9.225.3:34623)
0809 00:57:51.645505 (+ 2508us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:51.645516 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:51.646128 (+ 612us) server_negotiation.cc:407] Connection header received
0809 00:57:51.646475 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:51.646485 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:51.646730 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:51.646904 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:51.648122 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.649460 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.652059 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.653591 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.653767 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.654053 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:51.654363 (+ 310us) server_negotiation.cc:299] Negotiation successful
0809 00:57:51.654561 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2304,"thread_start_us":2131,"threads_started":1}
I20260809 00:57:51.656239 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.642349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.9.225.2:36245)
0809 00:57:51.645055 (+ 2706us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:51.645070 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:51.645109 (+ 39us) server_negotiation.cc:407] Connection header received
0809 00:57:51.646038 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:51.646049 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:51.646363 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:51.647300 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:51.647558 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.649193 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.653174 (+ 3981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.654814 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.654957 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.655166 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:51.655728 (+ 562us) server_negotiation.cc:299] Negotiation successful
0809 00:57:51.655922 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2469,"thread_start_us":2356,"threads_started":1}
I20260809 00:57:51.657568 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.646315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:37955 (local address 127.0.0.1:42122)
0809 00:57:51.648245 (+ 1930us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:51.648265 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:51.648475 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:51.649412 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:51.649415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:51.649445 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:51.650056 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:51.650075 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.652026 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.652032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:51.654307 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:51.654323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.656680 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.656687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:51.656845 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.656877 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 00:57:51.656974 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 00:57:51.657122 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1794,"mutex_wait_us":42,"thread_start_us":1619,"threads_started":1}
I20260809 00:57:51.657616 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.646850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.9.225.1:37955)
0809 00:57:51.647633 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:51.647644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:51.648360 (+ 716us) server_negotiation.cc:407] Connection header received
0809 00:57:51.648671 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:51.648679 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:51.648916 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:51.649075 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:51.650275 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.651831 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.654546 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.656118 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.656230 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.656483 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:51.657068 (+ 585us) server_negotiation.cc:299] Negotiation successful
0809 00:57:51.657249 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":415,"threads_started":1}
I20260809 00:57:51.657836 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.642199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:36245 (local address 127.0.0.1:38146)
0809 00:57:51.643891 (+ 1692us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:51.643951 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:51.644290 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:51.646668 (+ 2378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:51.646674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:51.646701 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:51.647048 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:51.647057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.649385 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.649391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:51.652958 (+ 3567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:51.652974 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.655312 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.655319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:51.655507 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.655544 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 00:57:51.655646 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 00:57:51.655788 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1331,"mutex_wait_us":7,"thread_start_us":3499,"threads_started":1}
I20260809 00:57:51.660764 7661 tablet_service.cc:1467] Processing CreateTablet for tablet 30e20158f6f84fcd99c532a9fe8dcf57 (DEFAULT_TABLE table=test-table [id=206c0b47f28f476e89b61af165b64e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:57:51.661154 7443 tablet_service.cc:1467] Processing CreateTablet for tablet 30e20158f6f84fcd99c532a9fe8dcf57 (DEFAULT_TABLE table=test-table [id=206c0b47f28f476e89b61af165b64e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:57:51.662750 7443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e20158f6f84fcd99c532a9fe8dcf57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:51.662110 7208 tablet_service.cc:1467] Processing CreateTablet for tablet 30e20158f6f84fcd99c532a9fe8dcf57 (DEFAULT_TABLE table=test-table [id=206c0b47f28f476e89b61af165b64e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:57:51.662822 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e20158f6f84fcd99c532a9fe8dcf57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:51.663817 7208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e20158f6f84fcd99c532a9fe8dcf57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:51.675570 7806 tablet_bootstrap.cc:492] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424: Bootstrap starting.
I20260809 00:57:51.680410 7810 tablet_bootstrap.cc:492] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673: Bootstrap starting.
I20260809 00:57:51.680550 7806 tablet_bootstrap.cc:654] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:51.681490 7809 tablet_bootstrap.cc:492] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff: Bootstrap starting.
I20260809 00:57:51.682121 7806 log.cc:824] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:51.685431 7806 tablet_bootstrap.cc:492] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424: No bootstrap required, opened a new log
I20260809 00:57:51.685863 7806 ts_tablet_manager.cc:1397] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260809 00:57:51.686435 7809 tablet_bootstrap.cc:654] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:51.686686 7810 tablet_bootstrap.cc:654] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:51.688243 7810 log.cc:824] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:51.688336 7809 log.cc:824] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:51.692615 7810 tablet_bootstrap.cc:492] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673: No bootstrap required, opened a new log
I20260809 00:57:51.692679 7809 tablet_bootstrap.cc:492] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff: No bootstrap required, opened a new log
I20260809 00:57:51.693012 7810 ts_tablet_manager.cc:1397] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260809 00:57:51.693012 7809 ts_tablet_manager.cc:1397] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260809 00:57:51.695755 7806 raft_consensus.cc:348] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3593e08a424e5c9771486c60ed3424" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36245 } } peers { permanent_uuid: "ad27bb74ac09448daac545370e1f60ff" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34623 } } peers { permanent_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 37955 } }
I20260809 00:57:51.696273 7806 raft_consensus.cc:374] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:51.696431 7806 raft_consensus.cc:729] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3593e08a424e5c9771486c60ed3424, State: Initialized, Role: FOLLOWER
I20260809 00:57:51.697461 7806 consensus_queue.cc:260] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3593e08a424e5c9771486c60ed3424" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36245 } } peers { permanent_uuid: "ad27bb74ac09448daac545370e1f60ff" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34623 } } peers { permanent_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 37955 } }
I20260809 00:57:51.702364 7548 heartbeater.cc:503] Master 127.9.225.62:43477 was elected leader, sending a full tablet report...
I20260809 00:57:51.703161 7806 ts_tablet_manager.cc:1428] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260809 00:57:51.704850 7809 raft_consensus.cc:348] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3593e08a424e5c9771486c60ed3424" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36245 } } peers { permanent_uuid: "ad27bb74ac09448daac545370e1f60ff" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34623 } } peers { permanent_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 37955 } }
I20260809 00:57:51.705395 7809 raft_consensus.cc:374] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:51.705510 7809 raft_consensus.cc:729] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad27bb74ac09448daac545370e1f60ff, State: Initialized, Role: FOLLOWER
I20260809 00:57:51.705557 7810 raft_consensus.cc:348] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3593e08a424e5c9771486c60ed3424" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36245 } } peers { permanent_uuid: "ad27bb74ac09448daac545370e1f60ff" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34623 } } peers { permanent_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 37955 } }
I20260809 00:57:51.706087 7810 raft_consensus.cc:374] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:51.706209 7810 raft_consensus.cc:729] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6b0c93bd8848aaa1e3ce092cbfa673, State: Initialized, Role: FOLLOWER
I20260809 00:57:51.706437 7809 consensus_queue.cc:260] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3593e08a424e5c9771486c60ed3424" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36245 } } peers { permanent_uuid: "ad27bb74ac09448daac545370e1f60ff" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34623 } } peers { permanent_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 37955 } }
I20260809 00:57:51.706854 7810 consensus_queue.cc:260] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3593e08a424e5c9771486c60ed3424" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36245 } } peers { permanent_uuid: "ad27bb74ac09448daac545370e1f60ff" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34623 } } peers { permanent_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 37955 } }
I20260809 00:57:51.709070 7809 ts_tablet_manager.cc:1428] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260809 00:57:51.709168 7765 heartbeater.cc:503] Master 127.9.225.62:43477 was elected leader, sending a full tablet report...
I20260809 00:57:51.713956 7810 ts_tablet_manager.cc:1428] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673: Time spent starting tablet: real 0.021s user 0.017s sys 0.005s
I20260809 00:57:51.725658 7821 raft_consensus.cc:482] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:57:51.726054 7821 raft_consensus.cc:504] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3593e08a424e5c9771486c60ed3424" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36245 } } peers { permanent_uuid: "ad27bb74ac09448daac545370e1f60ff" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34623 } } peers { permanent_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 37955 } }
I20260809 00:57:51.728117 7821 leader_election.cc:288] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd3593e08a424e5c9771486c60ed3424 (127.9.225.2:36245), ad6b0c93bd8848aaa1e3ce092cbfa673 (127.9.225.1:37955)
I20260809 00:57:51.736156 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.728941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:43109 (local address 127.9.225.2:36245)
0809 00:57:51.729167 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:51.729174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:51.729199 (+ 25us) server_negotiation.cc:407] Connection header received
0809 00:57:51.729329 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:51.729360 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:51.729471 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:51.729582 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:51.730125 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.731217 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.733681 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.735092 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.735275 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.735392 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:51.735739 (+ 347us) server_negotiation.cc:299] Negotiation successful
0809 00:57:51.735892 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 00:57:51.736239 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.728589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:36245 (local address 127.9.225.3:43109)
0809 00:57:51.729041 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:51.729063 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:51.729194 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:51.729629 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:51.729637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:51.729670 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:51.729906 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:51.729919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.731381 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.731385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:51.733319 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:51.733373 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.735347 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.735355 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:51.735530 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.735559 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 00:57:51.735690 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 00:57:51.735866 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":1}
I20260809 00:57:51.736653 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.728595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:51975 (local address 127.9.225.1:37955)
0809 00:57:51.728855 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:51.728864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:51.729915 (+ 1051us) server_negotiation.cc:407] Connection header received
0809 00:57:51.730142 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:51.730149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:51.730244 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:51.730385 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:51.730976 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.731906 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.734155 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.735535 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.735699 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.735808 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:51.736267 (+ 459us) server_negotiation.cc:299] Negotiation successful
0809 00:57:51.736423 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 00:57:51.736945 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.728953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:37955 (local address 127.9.225.3:51975)
0809 00:57:51.729786 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:51.729804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:51.729925 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:51.730407 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:51.730410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:51.730442 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:51.730783 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:51.730791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.732086 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.732092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:51.733971 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:51.733985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.735776 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.735782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:51.735943 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.735975 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 00:57:51.736225 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 00:57:51.736398 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":501,"threads_started":1}
I20260809 00:57:51.738018 7473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e20158f6f84fcd99c532a9fe8dcf57" candidate_uuid: "ad27bb74ac09448daac545370e1f60ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3593e08a424e5c9771486c60ed3424" is_pre_election: true
I20260809 00:57:51.738042 7243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e20158f6f84fcd99c532a9fe8dcf57" candidate_uuid: "ad27bb74ac09448daac545370e1f60ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673" is_pre_election: true
I20260809 00:57:51.738710 7243 raft_consensus.cc:2454] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad27bb74ac09448daac545370e1f60ff in term 0.
I20260809 00:57:51.738699 7473 raft_consensus.cc:2454] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad27bb74ac09448daac545370e1f60ff in term 0.
I20260809 00:57:51.739588 7583 leader_election.cc:302] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad27bb74ac09448daac545370e1f60ff, ad6b0c93bd8848aaa1e3ce092cbfa673; no voters:
I20260809 00:57:51.740146 7821 raft_consensus.cc:2781] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:57:51.740336 7821 raft_consensus.cc:482] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:57:51.740475 7821 raft_consensus.cc:3037] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:51.743032 7821 raft_consensus.cc:504] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3593e08a424e5c9771486c60ed3424" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36245 } } peers { permanent_uuid: "ad27bb74ac09448daac545370e1f60ff" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34623 } } peers { permanent_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 37955 } }
I20260809 00:57:51.743865 7821 leader_election.cc:288] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [CANDIDATE]: Term 1 election: Requested vote from peers fd3593e08a424e5c9771486c60ed3424 (127.9.225.2:36245), ad6b0c93bd8848aaa1e3ce092cbfa673 (127.9.225.1:37955)
I20260809 00:57:51.744549 7473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e20158f6f84fcd99c532a9fe8dcf57" candidate_uuid: "ad27bb74ac09448daac545370e1f60ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3593e08a424e5c9771486c60ed3424"
I20260809 00:57:51.744555 7243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e20158f6f84fcd99c532a9fe8dcf57" candidate_uuid: "ad27bb74ac09448daac545370e1f60ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673"
I20260809 00:57:51.744836 7473 raft_consensus.cc:3037] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:51.744828 7243 raft_consensus.cc:3037] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:51.747398 7243 raft_consensus.cc:2454] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad27bb74ac09448daac545370e1f60ff in term 1.
I20260809 00:57:51.747448 7473 raft_consensus.cc:2454] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad27bb74ac09448daac545370e1f60ff in term 1.
I20260809 00:57:51.748183 7581 leader_election.cc:302] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad27bb74ac09448daac545370e1f60ff, fd3593e08a424e5c9771486c60ed3424; no voters:
I20260809 00:57:51.748608 7821 raft_consensus.cc:2781] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:57:51.749192 7821 raft_consensus.cc:686] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 1 LEADER]: Becoming Leader. State: Replica: ad27bb74ac09448daac545370e1f60ff, State: Running, Role: LEADER
I20260809 00:57:51.749797 7821 consensus_queue.cc:237] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3593e08a424e5c9771486c60ed3424" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36245 } } peers { permanent_uuid: "ad27bb74ac09448daac545370e1f60ff" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34623 } } peers { permanent_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 37955 } }
I20260809 00:57:51.755517 7014 catalog_manager.cc:5495] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff reported cstate change: term changed from 0 to 1, leader changed from <none> to ad27bb74ac09448daac545370e1f60ff (127.9.225.3). New cstate: current_term: 1 leader_uuid: "ad27bb74ac09448daac545370e1f60ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3593e08a424e5c9771486c60ed3424" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad27bb74ac09448daac545370e1f60ff" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 34623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 37955 } health_report { overall_health: UNKNOWN } } }
W20260809 00:57:51.781795 7329 tablet.cc:2369] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:57:51.788393 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.782103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.9.225.3:34623)
0809 00:57:51.782302 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:51.782309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:51.782512 (+ 203us) server_negotiation.cc:407] Connection header received
0809 00:57:51.782782 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:51.782787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:51.782874 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:51.783075 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:57:51.783566 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.784313 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.785863 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.786291 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.786440 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.787170 (+ 730us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:57:51.787727 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:57:51.787950 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:51.788071 (+ 121us) server_negotiation.cc:299] Negotiation successful
0809 00:57:51.788125 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 00:57:51.799500 7473 raft_consensus.cc:1261] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424 [term 1 FOLLOWER]: Refusing update from remote peer ad27bb74ac09448daac545370e1f60ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:57:51.799500 7243 raft_consensus.cc:1261] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673 [term 1 FOLLOWER]: Refusing update from remote peer ad27bb74ac09448daac545370e1f60ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:57:51.800896 7826 consensus_queue.cc:1035] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6b0c93bd8848aaa1e3ce092cbfa673" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:57:51.801414 7821 consensus_queue.cc:1035] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd3593e08a424e5c9771486c60ed3424" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 00:57:51.808163 7766 tablet.cc:2369] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:57:51.822425 7832 mvcc.cc:205] Tried to move back new op lower bound from 7316427046076702720 to 7316427045891571712. Current Snapshot: MvccSnapshot[applied={T|T < 7316427046076702720 or (T in {7316427046076702720})}]
I20260809 00:57:51.822132 7833 mvcc.cc:205] Tried to move back new op lower bound from 7316427046076702720 to 7316427045891571712. Current Snapshot: MvccSnapshot[applied={T|T < 7316427046076702720}]
I20260809 00:57:51.832201 7013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:57:51.832680 7013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:57:51.837765 7013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dbf18182fe34b5b8c5bf9a3e5640282: Sending DeleteTablet for 3 replicas of tablet 30e20158f6f84fcd99c532a9fe8dcf57
I20260809 00:57:51.839196 7443 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e20158f6f84fcd99c532a9fe8dcf57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:57:51 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:38146
I20260809 00:57:51.839313 7208 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e20158f6f84fcd99c532a9fe8dcf57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:57:51 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:42122
I20260809 00:57:51.839324 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e20158f6f84fcd99c532a9fe8dcf57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:57:51 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:39790
I20260809 00:57:51.844547 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 7113
I20260809 00:57:51.854504 7843 tablet_replica.cc:330] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673: stopping tablet replica
I20260809 00:57:51.854785 7842 tablet_replica.cc:330] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424: stopping tablet replica
I20260809 00:57:51.855189 7842 raft_consensus.cc:2229] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:57:51.855189 7843 raft_consensus.cc:2229] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:57:51.855919 7842 raft_consensus.cc:2258] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:57:51.855919 7843 raft_consensus.cc:2258] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:57:51.859251 7842 ts_tablet_manager.cc:1905] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:57:51.861584 7844 tablet_replica.cc:330] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff: stopping tablet replica
I20260809 00:57:51.862130 7844 raft_consensus.cc:2229] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:57:51.863034 7844 raft_consensus.cc:2258] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:57:51.865435 7843 ts_tablet_manager.cc:1905] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:57:51.865527 7844 ts_tablet_manager.cc:1905] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:57:51.867780 7842 ts_tablet_manager.cc:1918] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:57:51.868009 7842 log.cc:1196] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/30e20158f6f84fcd99c532a9fe8dcf57
I20260809 00:57:51.868664 7842 ts_tablet_manager.cc:1939] T 30e20158f6f84fcd99c532a9fe8dcf57 P fd3593e08a424e5c9771486c60ed3424: Deleting consensus metadata
I20260809 00:57:51.871066 6989 catalog_manager.cc:4841] TS fd3593e08a424e5c9771486c60ed3424 (127.9.225.2:36245): tablet 30e20158f6f84fcd99c532a9fe8dcf57 (table test-table [id=206c0b47f28f476e89b61af165b64e8b]) successfully deleted
I20260809 00:57:51.874148 7844 ts_tablet_manager.cc:1918] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:57:51.874387 7844 log.cc:1196] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/30e20158f6f84fcd99c532a9fe8dcf57
I20260809 00:57:51.875046 7844 ts_tablet_manager.cc:1939] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad27bb74ac09448daac545370e1f60ff: Deleting consensus metadata
I20260809 00:57:51.876225 7843 ts_tablet_manager.cc:1918] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:57:51.876549 7843 log.cc:1196] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/30e20158f6f84fcd99c532a9fe8dcf57
I20260809 00:57:51.877282 7843 ts_tablet_manager.cc:1939] T 30e20158f6f84fcd99c532a9fe8dcf57 P ad6b0c93bd8848aaa1e3ce092cbfa673: Deleting consensus metadata
I20260809 00:57:51.877398 6987 catalog_manager.cc:4841] TS ad27bb74ac09448daac545370e1f60ff (127.9.225.3:34623): tablet 30e20158f6f84fcd99c532a9fe8dcf57 (table test-table [id=206c0b47f28f476e89b61af165b64e8b]) successfully deleted
Aug 09 00:57:51 dist-test-slave-j2vl krb5kdc[6949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:57:51.879410 6992 catalog_manager.cc:4841] TS ad6b0c93bd8848aaa1e3ce092cbfa673 (127.9.225.1:37955): tablet 30e20158f6f84fcd99c532a9fe8dcf57 (table test-table [id=206c0b47f28f476e89b61af165b64e8b]) successfully deleted
I20260809 00:57:51.887599 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:51.846160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.9.225.1:37955)
0809 00:57:51.846385 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:51.846394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:51.860290 (+ 13896us) server_negotiation.cc:407] Connection header received
0809 00:57:51.860900 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:51.860907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:51.861010 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:51.861093 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:51.869587 (+ 8494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.870783 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.872907 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:51.873631 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:51.873758 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:51.880402 (+ 6644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:51.880416 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:51.880432 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:51.880472 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:51.884052 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:51.885109 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:51.885118 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:51.885120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:51.885274 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:51.885788 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:51.885792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:51.885800 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:51.886213 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:51.886459 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:51.886952 (+ 493us) server_negotiation.cc:299] Negotiation successful
0809 00:57:51.887163 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 00:57:51.888577 7317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:52.002409 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 7113
I20260809 00:57:52.032217 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 7336
Aug 09 00:57:52 dist-test-slave-j2vl krb5kdc[6949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:57:52.050899 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:52.033213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.9.225.2:36245)
0809 00:57:52.033449 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:52.033456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:52.033479 (+ 23us) server_negotiation.cc:407] Connection header received
0809 00:57:52.033666 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:52.033672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:52.033810 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:52.033926 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:52.035760 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:52.036536 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:52.038203 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:52.038634 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:52.038787 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:52.044120 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:52.044141 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:52.044157 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:52.044213 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:52.047471 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:52.048404 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:52.048413 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:52.048415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:52.048558 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:52.048996 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:52.049001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:52.049003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:52.049567 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:52.049917 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:52.050262 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 00:57:52.050523 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 00:57:52.051717 7541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:52.148160 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 7336
I20260809 00:57:52.176016 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 7558
Aug 09 00:57:52 dist-test-slave-j2vl krb5kdc[6949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:57:52.196481 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:52.176917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.9.225.3:34623)
0809 00:57:52.177117 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:52.177125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:52.177165 (+ 40us) server_negotiation.cc:407] Connection header received
0809 00:57:52.177409 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:52.177416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:52.177501 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:52.177628 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:52.179110 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:52.180007 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:52.181555 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:52.182117 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:52.182280 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:52.188079 (+ 5799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:52.188098 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:52.188118 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:52.188163 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:52.192055 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:52.193049 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:52.193061 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:52.193063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:52.193284 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:52.194095 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:52.194105 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:52.194111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:52.194833 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:52.195247 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:52.195714 (+ 467us) server_negotiation.cc:299] Negotiation successful
0809 00:57:52.196015 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 00:57:52.197513 7757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:52.328825 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 7558
I20260809 00:57:52.362128 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 6965
I20260809 00:57:52.363449 7058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:52.485570 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 6965
2026-08-09T00:57:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5298 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T00:57:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:57:52Z Disabled control of system clock
I20260809 00:57:52.599006 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35275
--webserver_interface=127.9.225.62
--webserver_port=0
--builtin_ntp_servers=127.9.225.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:35275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 00:57:52.996250 7854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:52.996601 7854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:52.996743 7854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:53.004501 7854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:57:53.004686 7854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:53.004778 7854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:53.004889 7854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:57:53.004937 7854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:57:53.015535 7854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:35275
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:53.017081 7854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:53.019174 7854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:53.032573 7866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:53.032661 7863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:53.032670 7864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:53.038978 7854 server_base.cc:1034] running on GCE node
I20260809 00:57:53.040092 7854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:53.041913 7854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:53.043190 7854 hybrid_clock.cc:648] HybridClock initialized: now 1786237073043149 us; error 47 us; skew 500 ppm
I20260809 00:57:53.043609 7854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:57:53.046433 7854 webserver.cc:460] Webserver started at http://127.9.225.62:38831/ using document root <none> and password file <none>
I20260809 00:57:53.047572 7854 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:53.047755 7854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:53.048147 7854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:53.051633 7854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "8b1b757f26fa4b49b5d0984701fae527"
format_stamp: "Formatted at 2026-08-09 00:57:53 on dist-test-slave-j2vl"
server_key: "51278a71c2e2a6863b53031f799784a9"
server_key_iv: "7c8b4865ec676fb8d33d023d7e71d040"
server_key_version: "encryptionkey@0"
I20260809 00:57:53.052647 7854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "8b1b757f26fa4b49b5d0984701fae527"
format_stamp: "Formatted at 2026-08-09 00:57:53 on dist-test-slave-j2vl"
server_key: "51278a71c2e2a6863b53031f799784a9"
server_key_iv: "7c8b4865ec676fb8d33d023d7e71d040"
server_key_version: "encryptionkey@0"
I20260809 00:57:53.058879 7854 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 00:57:53.063095 7874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:53.064653 7854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 00:57:53.064913 7854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "8b1b757f26fa4b49b5d0984701fae527"
format_stamp: "Formatted at 2026-08-09 00:57:53 on dist-test-slave-j2vl"
server_key: "51278a71c2e2a6863b53031f799784a9"
server_key_iv: "7c8b4865ec676fb8d33d023d7e71d040"
server_key_version: "encryptionkey@0"
I20260809 00:57:53.065188 7854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:53.084556 7854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:53.085707 7854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:53.086055 7854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:53.108238 7854 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:35275
I20260809 00:57:53.108248 7961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:35275 every 8 connection(s)
I20260809 00:57:53.109901 7854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:57:53.114647 7962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:53.117479 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 7854
I20260809 00:57:53.117717 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:57:53.118373 10116 external_mini_cluster.cc:1428] Setting key 7b0da05be8c88cac1179293553bdae83
I20260809 00:57:53.129052 7962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527: Bootstrap starting.
I20260809 00:57:53.132843 7962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:53.134232 7962 log.cc:824] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:53.134549 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:53.120711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.9.225.62:35275)
0809 00:57:53.122238 (+ 1527us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:53.122258 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:53.122328 (+ 70us) server_negotiation.cc:407] Connection header received
0809 00:57:53.124238 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:53.124265 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:53.124808 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:53.125231 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:53.125960 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:53.127388 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:53.128852 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:53.129812 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:53.130043 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:53.131588 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:53.131604 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:53.131623 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:53.132261 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:53.132369 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:53.132394 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:53.132566 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:53.132848 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 00:57:53.133285 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":512,"threads_started":1}
I20260809 00:57:53.137960 7962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527: No bootstrap required, opened a new log
I20260809 00:57:53.146081 7962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1b757f26fa4b49b5d0984701fae527" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35275 } }
I20260809 00:57:53.146587 7962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:53.146714 7962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1b757f26fa4b49b5d0984701fae527, State: Initialized, Role: FOLLOWER
I20260809 00:57:53.147380 7962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1b757f26fa4b49b5d0984701fae527" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35275 } }
I20260809 00:57:53.147656 7962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:57:53.147822 7962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:57:53.148026 7962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:53.150830 7962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1b757f26fa4b49b5d0984701fae527" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35275 } }
I20260809 00:57:53.151454 7962 leader_election.cc:302] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b1b757f26fa4b49b5d0984701fae527; no voters:
I20260809 00:57:53.152158 7962 leader_election.cc:288] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:57:53.152345 7970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:57:53.153131 7970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [term 1 LEADER]: Becoming Leader. State: Replica: 8b1b757f26fa4b49b5d0984701fae527, State: Running, Role: LEADER
I20260809 00:57:53.153888 7970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1b757f26fa4b49b5d0984701fae527" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35275 } }
I20260809 00:57:53.154593 7962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:57:53.157317 7975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b1b757f26fa4b49b5d0984701fae527. Latest consensus state: current_term: 1 leader_uuid: "8b1b757f26fa4b49b5d0984701fae527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1b757f26fa4b49b5d0984701fae527" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35275 } } }
I20260809 00:57:53.158037 7975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [sys.catalog]: This master's current role is: LEADER
I20260809 00:57:53.157317 7974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b1b757f26fa4b49b5d0984701fae527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1b757f26fa4b49b5d0984701fae527" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 35275 } } }
I20260809 00:57:53.158667 7974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527 [sys.catalog]: This master's current role is: LEADER
I20260809 00:57:53.159797 7984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:57:53.166054 7984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:57:53.176784 7984 catalog_manager.cc:1347] Generated new cluster ID: 80820649b9424a74902d2f2f7218ccfc
I20260809 00:57:53.176942 7984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:57:53.211205 7984 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:57:53.213260 7984 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:57:53.224946 7984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527: Generated new TSK 0
I20260809 00:57:53.226342 7984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:57:53.238435 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35275
--builtin_ntp_servers=127.9.225.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 00:57:53.632649 8000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:53.632982 8000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:53.633145 8000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:53.641139 8000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:53.641283 8000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:53.641485 8000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:57:53.651548 8000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:53.653121 8000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:53.655257 8000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:53.670804 8008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:53.670861 8009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:53.670907 8014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:53.673206 8000 server_base.cc:1034] running on GCE node
I20260809 00:57:53.674058 8000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:53.675282 8000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:53.676518 8000 hybrid_clock.cc:648] HybridClock initialized: now 1786237073676462 us; error 44 us; skew 500 ppm
I20260809 00:57:53.676944 8000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:57:53.680073 8000 webserver.cc:460] Webserver started at http://127.9.225.1:34409/ using document root <none> and password file <none>
I20260809 00:57:53.681025 8000 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:53.681182 8000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:53.681674 8000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:53.685303 8000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "1260a03f4a2148e9a3db74cbfdfce74c"
format_stamp: "Formatted at 2026-08-09 00:57:53 on dist-test-slave-j2vl"
server_key: "44268df158b0aae1d3249756844b1b81"
server_key_iv: "c8d51792d249fe04d945d09a01178e03"
server_key_version: "encryptionkey@0"
I20260809 00:57:53.686416 8000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "1260a03f4a2148e9a3db74cbfdfce74c"
format_stamp: "Formatted at 2026-08-09 00:57:53 on dist-test-slave-j2vl"
server_key: "44268df158b0aae1d3249756844b1b81"
server_key_iv: "c8d51792d249fe04d945d09a01178e03"
server_key_version: "encryptionkey@0"
I20260809 00:57:53.693914 8000 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260809 00:57:53.698590 8025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:53.700220 8000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:57:53.700469 8000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "1260a03f4a2148e9a3db74cbfdfce74c"
format_stamp: "Formatted at 2026-08-09 00:57:53 on dist-test-slave-j2vl"
server_key: "44268df158b0aae1d3249756844b1b81"
server_key_iv: "c8d51792d249fe04d945d09a01178e03"
server_key_version: "encryptionkey@0"
I20260809 00:57:53.700742 8000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:53.726831 8000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:53.727828 8000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:53.728226 8000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:53.729511 8000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:53.731244 8000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:53.731401 8000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:53.731549 8000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:53.731642 8000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:53.769675 8000 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:35837
I20260809 00:57:53.769742 8204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:35837 every 8 connection(s)
I20260809 00:57:53.771272 8000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:57:53.777812 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 8000
I20260809 00:57:53.778031 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:57:53.778539 10116 external_mini_cluster.cc:1428] Setting key 6e0ca7db729a80cbf90ebd7cae6131ab
I20260809 00:57:53.781644 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35275
--builtin_ntp_servers=127.9.225.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 00:57:53.785665 8211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:53.774280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:38289 (local address 127.9.225.62:35275)
0809 00:57:53.774819 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:53.774824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:53.776012 (+ 1188us) server_negotiation.cc:407] Connection header received
0809 00:57:53.778192 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:53.778199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:53.778311 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:53.778440 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:53.779781 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:53.780531 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:53.782422 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:53.782853 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:53.783027 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:53.784656 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:53.784663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:53.784666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:53.784688 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:53.784734 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:53.784746 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:53.784853 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:53.785111 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 00:57:53.785236 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":281,"threads_started":1}
I20260809 00:57:53.786831 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:53.774812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35275 (local address 127.9.225.1:38289)
0809 00:57:53.775802 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:53.775868 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:53.777818 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:53.778781 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:53.778792 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:53.778902 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:53.779588 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:53.779608 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:53.780711 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:53.780715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:53.782250 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:53.782267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:53.783146 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:53.783155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:53.783424 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:53.784244 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:57:53.784246 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:53.784295 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:57:53.784393 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:53.784852 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:53.784868 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:53.784879 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 00:57:53.785097 (+ 218us) client_negotiation.cc:240] Negotiation successful
0809 00:57:53.785616 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":299,"threads_started":1}
I20260809 00:57:53.788748 8205 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35275
I20260809 00:57:53.789101 8205 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:53.790032 8205 heartbeater.cc:511] Master 127.9.225.62:35275 requested a full tablet report, sending...
I20260809 00:57:53.792691 7902 ts_manager.cc:194] Registered new tserver with Master: 1260a03f4a2148e9a3db74cbfdfce74c (127.9.225.1:35837)
I20260809 00:57:53.795051 7902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.225.1:38289
W20260809 00:57:54.178336 8214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:54.178750 8214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:54.178918 8214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:54.186820 8214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:54.186997 8214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:54.187186 8214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:57:54.196956 8214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:54.198482 8214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:54.200487 8214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:54.215765 8221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:54.215806 8227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:54.216156 8220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:54.216660 8214 server_base.cc:1034] running on GCE node
I20260809 00:57:54.217667 8214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:54.218792 8214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:54.220062 8214 hybrid_clock.cc:648] HybridClock initialized: now 1786237074219993 us; error 73 us; skew 500 ppm
I20260809 00:57:54.220467 8214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:57:54.223258 8214 webserver.cc:460] Webserver started at http://127.9.225.2:39543/ using document root <none> and password file <none>
I20260809 00:57:54.224124 8214 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:54.224282 8214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:54.224651 8214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:54.228143 8214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "b1ef2ec720d54583b2ce419c05723a8e"
format_stamp: "Formatted at 2026-08-09 00:57:54 on dist-test-slave-j2vl"
server_key: "9924c6bc6f163b78dee23a390fffe6cd"
server_key_iv: "efdec03074728dbaef277637cd489556"
server_key_version: "encryptionkey@0"
I20260809 00:57:54.229176 8214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "b1ef2ec720d54583b2ce419c05723a8e"
format_stamp: "Formatted at 2026-08-09 00:57:54 on dist-test-slave-j2vl"
server_key: "9924c6bc6f163b78dee23a390fffe6cd"
server_key_iv: "efdec03074728dbaef277637cd489556"
server_key_version: "encryptionkey@0"
I20260809 00:57:54.235575 8214 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260809 00:57:54.239850 8237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:54.241546 8214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 00:57:54.241770 8214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "b1ef2ec720d54583b2ce419c05723a8e"
format_stamp: "Formatted at 2026-08-09 00:57:54 on dist-test-slave-j2vl"
server_key: "9924c6bc6f163b78dee23a390fffe6cd"
server_key_iv: "efdec03074728dbaef277637cd489556"
server_key_version: "encryptionkey@0"
I20260809 00:57:54.242034 8214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:54.259859 8214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:54.260926 8214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:54.261251 8214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:54.262517 8214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:54.264156 8214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:54.264251 8214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:54.264421 8214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:54.264518 8214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:54.300712 8214 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:37977
I20260809 00:57:54.300768 8407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:37977 every 8 connection(s)
I20260809 00:57:54.302408 8214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:57:54.311905 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 8214
I20260809 00:57:54.312119 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:57:54.312610 10116 external_mini_cluster.cc:1428] Setting key b30eec96453c1152f4c8101325d5cce7
I20260809 00:57:54.316237 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35275
--builtin_ntp_servers=127.9.225.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 00:57:54.317328 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:54.305655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:43311 (local address 127.9.225.62:35275)
0809 00:57:54.306116 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:54.306121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:54.307551 (+ 1430us) server_negotiation.cc:407] Connection header received
0809 00:57:54.309595 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:54.309602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:54.309711 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:54.309845 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:54.311094 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.311877 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.313861 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.314330 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.314484 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:54.316267 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:54.316273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:54.316277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:54.316297 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:54.316393 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:54.316417 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:54.316533 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:54.316830 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 00:57:54.316950 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":240,"threads_started":1}
I20260809 00:57:54.318642 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:54.306319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35275 (local address 127.9.225.2:43311)
0809 00:57:54.307353 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:54.307402 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:54.309201 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:54.310138 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:54.310149 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:54.310251 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:54.310892 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:54.310910 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.312058 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.312066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:54.313702 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:54.313716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.314577 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.314585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:54.314963 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:54.315802 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:57:54.315809 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:54.315876 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:57:54.316003 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:54.316566 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:54.316586 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:54.316594 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 00:57:54.316817 (+ 223us) client_negotiation.cc:240] Negotiation successful
0809 00:57:54.317311 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":305,"threads_started":1}
I20260809 00:57:54.320453 8409 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35275
I20260809 00:57:54.320818 8409 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:54.321817 8409 heartbeater.cc:511] Master 127.9.225.62:35275 requested a full tablet report, sending...
I20260809 00:57:54.323611 7902 ts_manager.cc:194] Registered new tserver with Master: b1ef2ec720d54583b2ce419c05723a8e (127.9.225.2:37977)
I20260809 00:57:54.324483 7902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.225.2:43311
W20260809 00:57:54.715943 8419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:54.716292 8419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:54.716385 8419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:54.724758 8419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:54.724879 8419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:54.724993 8419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:57:54.734969 8419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:35275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:54.736408 8419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:54.738837 8419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:54.754016 8427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:54.754631 8426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:54.754591 8430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:54.755299 8419 server_base.cc:1034] running on GCE node
I20260809 00:57:54.756217 8419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:54.757459 8419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:54.758702 8419 hybrid_clock.cc:648] HybridClock initialized: now 1786237074758619 us; error 72 us; skew 500 ppm
I20260809 00:57:54.759124 8419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:57:54.762248 8419 webserver.cc:460] Webserver started at http://127.9.225.3:44051/ using document root <none> and password file <none>
I20260809 00:57:54.763115 8419 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:54.763219 8419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:54.763625 8419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:54.767196 8419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "00a1e5a426fb44adadba1af255be3fb5"
format_stamp: "Formatted at 2026-08-09 00:57:54 on dist-test-slave-j2vl"
server_key: "50f65a7c5ef7e0edd4d3eb4cbd29a3c6"
server_key_iv: "fbc4a0ca27556748b926cb0f62f03167"
server_key_version: "encryptionkey@0"
I20260809 00:57:54.768193 8419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "00a1e5a426fb44adadba1af255be3fb5"
format_stamp: "Formatted at 2026-08-09 00:57:54 on dist-test-slave-j2vl"
server_key: "50f65a7c5ef7e0edd4d3eb4cbd29a3c6"
server_key_iv: "fbc4a0ca27556748b926cb0f62f03167"
server_key_version: "encryptionkey@0"
I20260809 00:57:54.775005 8419 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260809 00:57:54.779373 8440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:54.781054 8419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 00:57:54.781281 8419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "00a1e5a426fb44adadba1af255be3fb5"
format_stamp: "Formatted at 2026-08-09 00:57:54 on dist-test-slave-j2vl"
server_key: "50f65a7c5ef7e0edd4d3eb4cbd29a3c6"
server_key_iv: "fbc4a0ca27556748b926cb0f62f03167"
server_key_version: "encryptionkey@0"
I20260809 00:57:54.781623 8419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:54.799932 8205 heartbeater.cc:503] Master 127.9.225.62:35275 was elected leader, sending a full tablet report...
I20260809 00:57:54.804298 8419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:54.805402 8419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:54.805714 8419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:54.806890 8419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:54.808562 8419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:54.808655 8419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:54.808806 8419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:54.808892 8419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:54.844539 8419 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:33473
I20260809 00:57:54.844655 8623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:33473 every 8 connection(s)
I20260809 00:57:54.846282 8419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:57:54.856389 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 8419
I20260809 00:57:54.856623 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:57:54.857100 10116 external_mini_cluster.cc:1428] Setting key 7adc705674ddcac7fef9c166970389ec
I20260809 00:57:54.861490 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:54.849403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:44743 (local address 127.9.225.62:35275)
0809 00:57:54.850068 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:54.850074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:54.851300 (+ 1226us) server_negotiation.cc:407] Connection header received
0809 00:57:54.853254 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:54.853259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:54.853400 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:54.853527 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:54.854795 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.855817 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.857761 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.858251 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.858390 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:54.860405 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:54.860411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:54.860415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:54.860436 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:54.860489 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:54.860502 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:54.860627 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:54.860868 (+ 241us) server_negotiation.cc:299] Negotiation successful
0809 00:57:54.860997 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":364,"threads_started":1}
I20260809 00:57:54.862668 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:54.850014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:35275 (local address 127.9.225.3:44743)
0809 00:57:54.851125 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:54.851173 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:54.852924 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:54.853768 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:54.853778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:54.853882 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:54.854603 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:54.854622 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.855962 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.855966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:54.857565 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:54.857597 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.858535 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.858546 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:54.858899 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:54.859980 (+ 1081us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:57:54.859984 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:54.860051 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:57:54.860185 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:54.860628 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:54.860638 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:54.860646 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 00:57:54.860852 (+ 206us) client_negotiation.cc:240] Negotiation successful
0809 00:57:54.861353 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":334,"threads_started":1}
I20260809 00:57:54.864308 8624 heartbeater.cc:347] Connected to a master server at 127.9.225.62:35275
I20260809 00:57:54.864753 8624 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:54.865742 8624 heartbeater.cc:511] Master 127.9.225.62:35275 requested a full tablet report, sending...
I20260809 00:57:54.867447 7902 ts_manager.cc:194] Registered new tserver with Master: 00a1e5a426fb44adadba1af255be3fb5 (127.9.225.3:33473)
I20260809 00:57:54.868327 7902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.225.3:44743
I20260809 00:57:54.874078 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:57:54.885335 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:54.879026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.9.225.62:35275)
0809 00:57:54.879235 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:54.879239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:54.879762 (+ 523us) server_negotiation.cc:407] Connection header received
0809 00:57:54.879910 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:54.879914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:54.879992 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:54.880093 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:54.880638 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.881441 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.882901 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.883338 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.883443 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:54.884536 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:54.884541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:54.884544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:54.884564 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:54.884599 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:54.884610 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:54.884729 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:54.884978 (+ 249us) server_negotiation.cc:299] Negotiation successful
0809 00:57:54.885048 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 00:57:54.890249 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:57:54.897797 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:57:54.936475 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:54.911952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:37977 (local address 127.0.0.1:59646)
0809 00:57:54.925859 (+ 13907us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:54.925886 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:54.926063 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:54.927267 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:54.927276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:54.927331 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:54.928182 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:54.928192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.930275 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.930285 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:54.932740 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:54.932758 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.935269 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.935280 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:54.935589 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:54.935650 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 00:57:54.935795 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 00:57:54.936051 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13760,"mutex_wait_us":311,"spinlock_wait_cycles":7296,"thread_start_us":11526,"threads_started":1}
I20260809 00:57:54.936510 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:54.912080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.9.225.3:33473)
0809 00:57:54.926309 (+ 14229us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:54.926317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:54.926348 (+ 31us) server_negotiation.cc:407] Connection header received
0809 00:57:54.926450 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:54.926456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:54.926697 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:54.926932 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:54.928389 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.930086 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.933469 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.934682 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.934857 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:54.935160 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:54.935889 (+ 729us) server_negotiation.cc:299] Negotiation successful
0809 00:57:54.936110 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14037,"thread_start_us":13876,"threads_started":1}
I20260809 00:57:54.936524 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:54.912223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.9.225.2:37977)
0809 00:57:54.925431 (+ 13208us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:54.925457 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:54.926227 (+ 770us) server_negotiation.cc:407] Connection header received
0809 00:57:54.926366 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:54.926376 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:54.926698 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:54.926932 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:54.928377 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.930102 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.932975 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.934675 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.934844 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:54.935122 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:54.935990 (+ 868us) server_negotiation.cc:299] Negotiation successful
0809 00:57:54.936156 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13019,"thread_start_us":12884,"threads_started":1}
I20260809 00:57:54.937161 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:54.911693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:33473 (local address 127.0.0.1:58216)
0809 00:57:54.924196 (+ 12503us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:54.924291 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:54.924642 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:54.927267 (+ 2625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:54.927276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:54.927325 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:54.928091 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:54.928108 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.930275 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.930285 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:54.933204 (+ 2919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:54.933217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.935316 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.935326 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:54.935597 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:54.935666 (+ 69us) client_negotiation.cc:769] Sending connection context
0809 00:57:54.935799 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 00:57:54.936827 (+ 1028us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12003,"mutex_wait_us":874,"spinlock_wait_cycles":768,"thread_start_us":13843,"threads_started":1}
I20260809 00:57:54.938372 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:54.914207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:35837 (local address 127.0.0.1:42440)
0809 00:57:54.928054 (+ 13847us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:54.928085 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:54.928272 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:54.930105 (+ 1833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:54.930111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:54.930143 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:54.930987 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:54.931005 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.933141 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.933147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:54.935361 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:54.935375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.937536 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.937543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:54.937724 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:54.937767 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 00:57:54.937872 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 00:57:54.938008 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13679,"mutex_wait_us":45,"thread_start_us":13469,"threads_started":1}
I20260809 00:57:54.938987 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:54.913828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.9.225.1:35837)
0809 00:57:54.929053 (+ 15225us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:54.929060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:54.929090 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:57:54.929174 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:54.929180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:54.929486 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:54.929778 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:54.931271 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.932976 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.935613 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:54.937313 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:54.937498 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:54.937804 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:54.937919 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 00:57:54.938091 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15025,"thread_start_us":14859,"threads_started":1}
I20260809 00:57:54.941442 8306 tablet_service.cc:1467] Processing CreateTablet for tablet ef5b412fc735457c8be3d8614b3e1ae2 (DEFAULT_TABLE table=test-table [id=9bbbf141351c491b93a3cc0b0aac0a0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:57:54.941548 8510 tablet_service.cc:1467] Processing CreateTablet for tablet ef5b412fc735457c8be3d8614b3e1ae2 (DEFAULT_TABLE table=test-table [id=9bbbf141351c491b93a3cc0b0aac0a0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:57:54.942929 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef5b412fc735457c8be3d8614b3e1ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:54.943135 8510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef5b412fc735457c8be3d8614b3e1ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:54.943069 8099 tablet_service.cc:1467] Processing CreateTablet for tablet ef5b412fc735457c8be3d8614b3e1ae2 (DEFAULT_TABLE table=test-table [id=9bbbf141351c491b93a3cc0b0aac0a0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:57:54.944772 8099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef5b412fc735457c8be3d8614b3e1ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:54.956456 8654 tablet_bootstrap.cc:492] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e: Bootstrap starting.
I20260809 00:57:54.960942 8654 tablet_bootstrap.cc:654] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:54.962419 8654 log.cc:824] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:54.963039 8657 tablet_bootstrap.cc:492] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c: Bootstrap starting.
I20260809 00:57:54.965224 8656 tablet_bootstrap.cc:492] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5: Bootstrap starting.
I20260809 00:57:54.967571 8657 tablet_bootstrap.cc:654] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:54.968291 8654 tablet_bootstrap.cc:492] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e: No bootstrap required, opened a new log
I20260809 00:57:54.968701 8654 ts_tablet_manager.cc:1397] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260809 00:57:54.969318 8656 tablet_bootstrap.cc:654] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:54.970130 8657 log.cc:824] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:54.970867 8656 log.cc:824] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:54.974010 8657 tablet_bootstrap.cc:492] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c: No bootstrap required, opened a new log
I20260809 00:57:54.974354 8656 tablet_bootstrap.cc:492] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5: No bootstrap required, opened a new log
I20260809 00:57:54.974386 8657 ts_tablet_manager.cc:1397] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260809 00:57:54.974663 8656 ts_tablet_manager.cc:1397] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 00:57:54.977861 8654 raft_consensus.cc:348] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a1e5a426fb44adadba1af255be3fb5" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 33473 } } peers { permanent_uuid: "b1ef2ec720d54583b2ce419c05723a8e" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37977 } } peers { permanent_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 35837 } }
I20260809 00:57:54.978324 8654 raft_consensus.cc:374] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:54.978474 8654 raft_consensus.cc:729] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ef2ec720d54583b2ce419c05723a8e, State: Initialized, Role: FOLLOWER
I20260809 00:57:54.979183 8654 consensus_queue.cc:260] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a1e5a426fb44adadba1af255be3fb5" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 33473 } } peers { permanent_uuid: "b1ef2ec720d54583b2ce419c05723a8e" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37977 } } peers { permanent_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 35837 } }
I20260809 00:57:54.982303 8409 heartbeater.cc:503] Master 127.9.225.62:35275 was elected leader, sending a full tablet report...
I20260809 00:57:54.983121 8654 ts_tablet_manager.cc:1428] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260809 00:57:54.986291 8656 raft_consensus.cc:348] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a1e5a426fb44adadba1af255be3fb5" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 33473 } } peers { permanent_uuid: "b1ef2ec720d54583b2ce419c05723a8e" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37977 } } peers { permanent_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 35837 } }
I20260809 00:57:54.986294 8657 raft_consensus.cc:348] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a1e5a426fb44adadba1af255be3fb5" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 33473 } } peers { permanent_uuid: "b1ef2ec720d54583b2ce419c05723a8e" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37977 } } peers { permanent_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 35837 } }
I20260809 00:57:54.986819 8656 raft_consensus.cc:374] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:54.986819 8657 raft_consensus.cc:374] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:54.986933 8657 raft_consensus.cc:729] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1260a03f4a2148e9a3db74cbfdfce74c, State: Initialized, Role: FOLLOWER
I20260809 00:57:54.986932 8656 raft_consensus.cc:729] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a1e5a426fb44adadba1af255be3fb5, State: Initialized, Role: FOLLOWER
I20260809 00:57:54.987670 8657 consensus_queue.cc:260] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a1e5a426fb44adadba1af255be3fb5" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 33473 } } peers { permanent_uuid: "b1ef2ec720d54583b2ce419c05723a8e" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37977 } } peers { permanent_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 35837 } }
I20260809 00:57:54.987672 8656 consensus_queue.cc:260] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a1e5a426fb44adadba1af255be3fb5" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 33473 } } peers { permanent_uuid: "b1ef2ec720d54583b2ce419c05723a8e" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37977 } } peers { permanent_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 35837 } }
I20260809 00:57:54.990059 8657 ts_tablet_manager.cc:1428] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260809 00:57:54.990417 8624 heartbeater.cc:503] Master 127.9.225.62:35275 was elected leader, sending a full tablet report...
I20260809 00:57:54.991117 8656 ts_tablet_manager.cc:1428] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260809 00:57:55.015220 8665 raft_consensus.cc:482] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:57:55.015666 8665 raft_consensus.cc:504] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a1e5a426fb44adadba1af255be3fb5" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 33473 } } peers { permanent_uuid: "b1ef2ec720d54583b2ce419c05723a8e" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37977 } } peers { permanent_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 35837 } }
I20260809 00:57:55.017619 8665 leader_election.cc:288] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a1e5a426fb44adadba1af255be3fb5 (127.9.225.3:33473), b1ef2ec720d54583b2ce419c05723a8e (127.9.225.2:37977)
W20260809 00:57:55.024075 8207 tablet.cc:2369] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:57:55.026505 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.018153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:45331 (local address 127.9.225.3:33473)
0809 00:57:55.018360 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:55.018367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:55.019171 (+ 804us) server_negotiation.cc:407] Connection header received
0809 00:57:55.019855 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:55.019860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:55.019922 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:55.020018 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:55.020615 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.021637 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.024329 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.025145 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.025228 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:55.025328 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:55.026185 (+ 857us) server_negotiation.cc:299] Negotiation successful
0809 00:57:55.026296 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 00:57:55.026760 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.018009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:33473 (local address 127.9.225.1:45331)
0809 00:57:55.018996 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:55.019018 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:55.019254 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:55.020083 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:55.020094 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:55.020120 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:55.020461 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:55.020470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.021793 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.021799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:55.023927 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:55.023938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.025596 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.025608 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:55.025899 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:55.026043 (+ 144us) client_negotiation.cc:769] Sending connection context
0809 00:57:55.026144 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 00:57:55.026337 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":147,"spinlock_wait_cycles":41728,"thread_start_us":556,"threads_started":1}
I20260809 00:57:55.026893 8669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.018086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:37977 (local address 127.9.225.1:55715)
0809 00:57:55.019055 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:55.019072 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:55.019208 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:55.019683 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:55.019687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:55.019715 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:57:55.020047 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:55.020059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.021462 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.021466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:55.023918 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:55.023938 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.025622 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.025629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:55.025904 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:55.026039 (+ 135us) client_negotiation.cc:769] Sending connection context
0809 00:57:55.026144 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 00:57:55.026337 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":125,"thread_start_us":548,"threads_started":1}
I20260809 00:57:55.027204 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.018153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:55715 (local address 127.9.225.2:37977)
0809 00:57:55.018360 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:55.018367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:55.019172 (+ 805us) server_negotiation.cc:407] Connection header received
0809 00:57:55.019390 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:55.019398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:55.019512 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:55.019631 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:57:55.020239 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.021297 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.024235 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.025317 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.025494 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:55.025563 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:55.026782 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0809 00:57:55.026932 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 00:57:55.028083 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef5b412fc735457c8be3d8614b3e1ae2" candidate_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ef2ec720d54583b2ce419c05723a8e" is_pre_election: true
I20260809 00:57:55.028085 8547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef5b412fc735457c8be3d8614b3e1ae2" candidate_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a1e5a426fb44adadba1af255be3fb5" is_pre_election: true
I20260809 00:57:55.028656 8338 raft_consensus.cc:2454] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1260a03f4a2148e9a3db74cbfdfce74c in term 0.
I20260809 00:57:55.028673 8547 raft_consensus.cc:2454] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1260a03f4a2148e9a3db74cbfdfce74c in term 0.
I20260809 00:57:55.029520 8030 leader_election.cc:302] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a1e5a426fb44adadba1af255be3fb5, 1260a03f4a2148e9a3db74cbfdfce74c; no voters:
I20260809 00:57:55.029984 8665 raft_consensus.cc:2781] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:57:55.030148 8665 raft_consensus.cc:482] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:57:55.030272 8665 raft_consensus.cc:3037] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:55.032740 8665 raft_consensus.cc:504] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a1e5a426fb44adadba1af255be3fb5" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 33473 } } peers { permanent_uuid: "b1ef2ec720d54583b2ce419c05723a8e" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37977 } } peers { permanent_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 35837 } }
I20260809 00:57:55.033509 8665 leader_election.cc:288] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [CANDIDATE]: Term 1 election: Requested vote from peers 00a1e5a426fb44adadba1af255be3fb5 (127.9.225.3:33473), b1ef2ec720d54583b2ce419c05723a8e (127.9.225.2:37977)
I20260809 00:57:55.033978 8547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef5b412fc735457c8be3d8614b3e1ae2" candidate_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a1e5a426fb44adadba1af255be3fb5"
I20260809 00:57:55.034096 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef5b412fc735457c8be3d8614b3e1ae2" candidate_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ef2ec720d54583b2ce419c05723a8e"
I20260809 00:57:55.034222 8547 raft_consensus.cc:3037] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:55.034317 8338 raft_consensus.cc:3037] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:55.037741 8547 raft_consensus.cc:2454] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1260a03f4a2148e9a3db74cbfdfce74c in term 1.
I20260809 00:57:55.037736 8338 raft_consensus.cc:2454] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1260a03f4a2148e9a3db74cbfdfce74c in term 1.
I20260809 00:57:55.038245 8027 leader_election.cc:302] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1260a03f4a2148e9a3db74cbfdfce74c, b1ef2ec720d54583b2ce419c05723a8e; no voters:
I20260809 00:57:55.038619 8665 raft_consensus.cc:2781] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:57:55.039080 8665 raft_consensus.cc:686] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 1 LEADER]: Becoming Leader. State: Replica: 1260a03f4a2148e9a3db74cbfdfce74c, State: Running, Role: LEADER
I20260809 00:57:55.039611 8665 consensus_queue.cc:237] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a1e5a426fb44adadba1af255be3fb5" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 33473 } } peers { permanent_uuid: "b1ef2ec720d54583b2ce419c05723a8e" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37977 } } peers { permanent_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 35837 } }
I20260809 00:57:55.045047 7902 catalog_manager.cc:5495] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1260a03f4a2148e9a3db74cbfdfce74c (127.9.225.1). New cstate: current_term: 1 leader_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a1e5a426fb44adadba1af255be3fb5" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 33473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ef2ec720d54583b2ce419c05723a8e" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1260a03f4a2148e9a3db74cbfdfce74c" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 35837 } health_report { overall_health: HEALTHY } } }
W20260809 00:57:55.055109 8411 tablet.cc:2369] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:57:55.072431 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.065424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.9.225.1:35837)
0809 00:57:55.065624 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:55.065629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:55.065678 (+ 49us) server_negotiation.cc:407] Connection header received
0809 00:57:55.065789 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:55.065793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:55.065888 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:55.066077 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:57:55.066577 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.067337 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.069007 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.069564 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.069712 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:55.071001 (+ 1289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:57:55.071633 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:57:55.071818 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:55.072163 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 00:57:55.072208 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 00:57:55.082662 8547 raft_consensus.cc:1261] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5 [term 1 FOLLOWER]: Refusing update from remote peer 1260a03f4a2148e9a3db74cbfdfce74c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:57:55.082660 8338 raft_consensus.cc:1261] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e [term 1 FOLLOWER]: Refusing update from remote peer 1260a03f4a2148e9a3db74cbfdfce74c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:57:55.083958 8670 consensus_queue.cc:1035] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a1e5a426fb44adadba1af255be3fb5" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 33473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:57:55.084401 8665 consensus_queue.cc:1035] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ef2ec720d54583b2ce419c05723a8e" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 00:57:55.098580 8625 tablet.cc:2369] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:57:55.119270 7900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:57:55.119758 7900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:57:55.124219 7900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b1b757f26fa4b49b5d0984701fae527: Sending DeleteTablet for 3 replicas of tablet ef5b412fc735457c8be3d8614b3e1ae2
I20260809 00:57:55.125641 8306 tablet_service.cc:1514] Processing DeleteTablet for tablet ef5b412fc735457c8be3d8614b3e1ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:57:55 UTC) from {username='slave'} at 127.0.0.1:59646
I20260809 00:57:55.125641 8510 tablet_service.cc:1514] Processing DeleteTablet for tablet ef5b412fc735457c8be3d8614b3e1ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:57:55 UTC) from {username='slave'} at 127.0.0.1:58216
I20260809 00:57:55.127578 8099 tablet_service.cc:1514] Processing DeleteTablet for tablet ef5b412fc735457c8be3d8614b3e1ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:57:55 UTC) from {username='slave'} at 127.0.0.1:42440
I20260809 00:57:55.134770 8693 tablet_replica.cc:330] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c: stopping tablet replica
I20260809 00:57:55.135406 8693 raft_consensus.cc:2229] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:57:55.136226 8693 raft_consensus.cc:2258] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:57:55.136863 8690 tablet_replica.cc:330] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e: stopping tablet replica
I20260809 00:57:55.137717 8690 raft_consensus.cc:2229] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:57:55.139541 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 8000
I20260809 00:57:55.140235 8693 ts_tablet_manager.cc:1905] T ef5b412fc735457c8be3d8614b3e1ae2 P 1260a03f4a2148e9a3db74cbfdfce74c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:57:55.140849 8690 raft_consensus.cc:2258] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:57:55.143344 8690 ts_tablet_manager.cc:1905] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:57:55.144114 8691 tablet_replica.cc:330] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5: stopping tablet replica
I20260809 00:57:55.144645 8691 raft_consensus.cc:2229] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:57:55.145236 8691 raft_consensus.cc:2258] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:57:55.147851 8691 ts_tablet_manager.cc:1905] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:57:55.152158 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.141182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.9.225.1:35837)
0809 00:57:55.141514 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:55.141522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:55.142535 (+ 1013us) server_negotiation.cc:407] Connection header received
0809 00:57:55.142885 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:55.142894 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:55.143012 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:55.143094 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:55.143924 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.145272 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.147695 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.148331 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.148461 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:55.150150 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:55.150163 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:55.150183 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:55.151095 (+ 912us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:55.151184 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:55.151212 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:55.151315 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:55.151591 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 00:57:55.151736 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260809 00:57:55.153921 8196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:55.156513 8691 ts_tablet_manager.cc:1918] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:57:55.156718 8690 ts_tablet_manager.cc:1918] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:57:55.156982 8691 log.cc:1196] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/ef5b412fc735457c8be3d8614b3e1ae2
I20260809 00:57:55.156993 8690 log.cc:1196] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/ef5b412fc735457c8be3d8614b3e1ae2
I20260809 00:57:55.157696 8690 ts_tablet_manager.cc:1939] T ef5b412fc735457c8be3d8614b3e1ae2 P b1ef2ec720d54583b2ce419c05723a8e: Deleting consensus metadata
I20260809 00:57:55.157761 8691 ts_tablet_manager.cc:1939] T ef5b412fc735457c8be3d8614b3e1ae2 P 00a1e5a426fb44adadba1af255be3fb5: Deleting consensus metadata
I20260809 00:57:55.160166 7876 catalog_manager.cc:4841] TS b1ef2ec720d54583b2ce419c05723a8e (127.9.225.2:37977): tablet ef5b412fc735457c8be3d8614b3e1ae2 (table test-table [id=9bbbf141351c491b93a3cc0b0aac0a0b]) successfully deleted
I20260809 00:57:55.160740 7879 catalog_manager.cc:4841] TS 00a1e5a426fb44adadba1af255be3fb5 (127.9.225.3:33473): tablet ef5b412fc735457c8be3d8614b3e1ae2 (table test-table [id=9bbbf141351c491b93a3cc0b0aac0a0b]) successfully deleted
I20260809 00:57:55.258821 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 8000
I20260809 00:57:55.292497 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 8214
W20260809 00:57:55.292482 7878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.225.1:35837 (error 108)
I20260809 00:57:55.294799 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.294251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:35837 (local address 127.0.0.1:49656)
0809 00:57:55.294458 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:55.294604 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260809 00:57:55.295094 7878 catalog_manager.cc:4584] TS 1260a03f4a2148e9a3db74cbfdfce74c (127.9.225.1:35837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef5b412fc735457c8be3d8614b3e1ae2: Network error: Client connection negotiation failed: client connection to 127.9.225.1:35837: connect: Connection refused (error 111)
I20260809 00:57:55.300630 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.293470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.9.225.2:37977)
0809 00:57:55.293660 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:55.293664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:55.293685 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:57:55.293930 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:55.293936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:55.294053 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:55.294172 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:55.294692 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.295669 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.297036 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.297570 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.297726 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:55.298984 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:55.298994 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:55.299012 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:55.299668 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:55.299720 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:55.299742 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:55.299881 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:55.300057 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 00:57:55.300228 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 00:57:55.301496 8401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:55.348937 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.348299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:35837 (local address 127.0.0.1:49664)
0809 00:57:55.348597 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:55.348720 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260809 00:57:55.350374 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.349901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:35837 (local address 127.0.0.1:49678)
0809 00:57:55.350088 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:55.350187 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 00:57:55.394622 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 8214
I20260809 00:57:55.411356 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.410676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:35837 (local address 127.0.0.1:49692)
0809 00:57:55.410974 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:55.411100 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260809 00:57:55.412690 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.412216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:35837 (local address 127.0.0.1:49694)
0809 00:57:55.412407 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:55.412514 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 00:57:55.426519 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 8419
I20260809 00:57:55.434419 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.427453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49170 (local address 127.9.225.3:33473)
0809 00:57:55.427635 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:55.427639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:55.427660 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:57:55.427812 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:55.427817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:55.427938 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:55.428038 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:55.428575 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.429531 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.430935 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:55.431366 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:55.431639 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:55.432775 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:55.432786 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:55.432803 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:57:55.433707 (+ 904us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:55.433770 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:57:55.433796 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:55.433905 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:55.434030 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 00:57:55.434147 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 00:57:55.435209 8618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:55.494081 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.493570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:35837 (local address 127.0.0.1:49710)
0809 00:57:55.493804 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:55.493899 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 00:57:55.495360 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.494869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:35837 (local address 127.0.0.1:49724)
0809 00:57:55.495150 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:55.495228 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260809 00:57:55.528178 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 8419
I20260809 00:57:55.559722 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 7854
I20260809 00:57:55.561056 7954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:57:55.677476 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:55.676757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:35837 (local address 127.0.0.1:49728)
0809 00:57:55.677015 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:55.677146 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 00:57:55.677999 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 7854
2026-08-09T00:57:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3195 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:57:55 dist-test-slave-j2vl krb5kdc[8707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:57:55 dist-test-slave-j2vl krb5kdc[8707](info): set up 2 sockets
Aug 09 00:57:55 dist-test-slave-j2vl krb5kdc[8707](info): commencing operation
krb5kdc: starting...
W20260809 00:57:57.102419 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.364s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:57:57 dist-test-slave-j2vl krb5kdc[8707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:57:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:57:57.469529 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:42371
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:42371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:57:57.857301 8729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:57.857791 8729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:57.857990 8729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:57.865869 8729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:57:57.866010 8729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:57.866089 8729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:57.866178 8729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:57:57.866247 8729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:57:57.877126 8729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:42371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:42371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:57.879329 8729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:57.880899 8729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:57.894659 8737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:57.894685 8742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:57.894585 8746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:57.895712 8729 server_base.cc:1034] running on GCE node
I20260809 00:57:57.896795 8729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:57.898739 8729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:57.900041 8729 hybrid_clock.cc:648] HybridClock initialized: now 1786237077900008 us; error 66 us; skew 500 ppm
I20260809 00:57:57.900592 8729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:57 dist-test-slave-j2vl krb5kdc[8707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237077, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:57.909554 8729 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:57:57.911669 8729 webserver.cc:460] Webserver started at http://127.9.225.62:40703/ using document root <none> and password file <none>
I20260809 00:57:57.912559 8729 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:57.912750 8729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:57.913141 8729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:57.916885 8729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "b1557df8496441f4b38ec3781f1cfecd"
format_stamp: "Formatted at 2026-08-09 00:57:57 on dist-test-slave-j2vl"
server_key: "5f19fde4ee83ba7d0c85dacd8c23a98c"
server_key_iv: "67d6735b635ef3a34b228e57a35406e8"
server_key_version: "encryptionkey@0"
I20260809 00:57:57.917881 8729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "b1557df8496441f4b38ec3781f1cfecd"
format_stamp: "Formatted at 2026-08-09 00:57:57 on dist-test-slave-j2vl"
server_key: "5f19fde4ee83ba7d0c85dacd8c23a98c"
server_key_iv: "67d6735b635ef3a34b228e57a35406e8"
server_key_version: "encryptionkey@0"
I20260809 00:57:57.924185 8729 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 00:57:57.928575 8760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:57.930368 8729 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 00:57:57.930606 8729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "b1557df8496441f4b38ec3781f1cfecd"
format_stamp: "Formatted at 2026-08-09 00:57:57 on dist-test-slave-j2vl"
server_key: "5f19fde4ee83ba7d0c85dacd8c23a98c"
server_key_iv: "67d6735b635ef3a34b228e57a35406e8"
server_key_version: "encryptionkey@0"
I20260809 00:57:57.930876 8729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:57.965932 8729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:57.971020 8729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:57.971378 8729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:57.992889 8729 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:42371
I20260809 00:57:57.992940 8844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:42371 every 8 connection(s)
I20260809 00:57:57.994547 8729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:57:57.998790 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 8729
I20260809 00:57:57.999020 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:57:57.999605 8845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:57:57.999683 10116 external_mini_cluster.cc:1428] Setting key 7533d7cec4a9905726aff0e7a60983a6
I20260809 00:57:58.016629 8845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd: Bootstrap starting.
Aug 09 00:57:58 dist-test-slave-j2vl krb5kdc[8707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:57:58.020440 8845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd: Neither blocks nor log segments found. Creating new log.
I20260809 00:57:58.021754 8845 log.cc:824] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd: Log is configured to *not* fsync() on all Append() calls
I20260809 00:57:58.025100 8845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd: No bootstrap required, opened a new log
I20260809 00:57:58.027016 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:58.001873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.9.225.62:42371)
0809 00:57:58.003017 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:58.003034 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:58.003087 (+ 53us) server_negotiation.cc:407] Connection header received
0809 00:57:58.005090 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:58.005119 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:58.005683 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:58.006097 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:58.008354 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:58.009830 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:58.011739 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:58.012546 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:58.012709 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:58.018601 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:58.018611 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:58.018629 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:58.018682 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:58.022339 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:58.023227 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:58.023236 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:58.023245 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:58.023449 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:58.023898 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:58.023905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:58.023913 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:58.024523 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:58.024884 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:58.025435 (+ 551us) server_negotiation.cc:299] Negotiation successful
0809 00:57:58.025938 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":582,"threads_started":1}
I20260809 00:57:58.034525 8845 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1557df8496441f4b38ec3781f1cfecd" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42371 } }
I20260809 00:57:58.035125 8845 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:57:58.035267 8845 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1557df8496441f4b38ec3781f1cfecd, State: Initialized, Role: FOLLOWER
I20260809 00:57:58.035996 8845 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1557df8496441f4b38ec3781f1cfecd" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42371 } }
I20260809 00:57:58.036283 8845 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:57:58.036444 8845 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:57:58.036643 8845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:57:58.039153 8845 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1557df8496441f4b38ec3781f1cfecd" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42371 } }
I20260809 00:57:58.039851 8845 leader_election.cc:302] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1557df8496441f4b38ec3781f1cfecd; no voters:
I20260809 00:57:58.040735 8845 leader_election.cc:288] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:57:58.040915 8854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:57:58.042109 8854 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [term 1 LEADER]: Becoming Leader. State: Replica: b1557df8496441f4b38ec3781f1cfecd, State: Running, Role: LEADER
I20260809 00:57:58.042784 8854 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1557df8496441f4b38ec3781f1cfecd" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42371 } }
I20260809 00:57:58.043412 8845 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:57:58.046027 8858 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1557df8496441f4b38ec3781f1cfecd. Latest consensus state: current_term: 1 leader_uuid: "b1557df8496441f4b38ec3781f1cfecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1557df8496441f4b38ec3781f1cfecd" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42371 } } }
I20260809 00:57:58.046466 8858 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [sys.catalog]: This master's current role is: LEADER
I20260809 00:57:58.046844 8856 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1557df8496441f4b38ec3781f1cfecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1557df8496441f4b38ec3781f1cfecd" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42371 } } }
I20260809 00:57:58.047336 8856 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd [sys.catalog]: This master's current role is: LEADER
I20260809 00:57:58.050221 8863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:57:58.055863 8863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:57:58.068490 8863 catalog_manager.cc:1347] Generated new cluster ID: 631efb2d80c740d88bc5d8256bd7e6ac
I20260809 00:57:58.068660 8863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:57:58.083911 8863 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:57:58.085466 8863 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:57:58.096755 8863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd: Generated new TSK 0
I20260809 00:57:58.097904 8863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.225.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
Entry for principal kudu/127.9.225.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab.
I20260809 00:57:58.194353 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42371
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:57:58.595495 8894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:58.595844 8894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:58.595949 8894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:58.604101 8894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:58.604246 8894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:58.604365 8894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:57:58.614950 8894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.1.keytab
--principal=kudu/127.9.225.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:58.616451 8894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:58.617952 8894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:58.633805 8900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:58.633780 8902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:58.633744 8908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:58.634709 8894 server_base.cc:1034] running on GCE node
I20260809 00:57:58.635465 8894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:58.636662 8894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:58.637933 8894 hybrid_clock.cc:648] HybridClock initialized: now 1786237078637859 us; error 70 us; skew 500 ppm
I20260809 00:57:58.638325 8894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:58 dist-test-slave-j2vl krb5kdc[8707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237078, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:58.641845 8894 init.cc:377] Logged in from keytab as kudu/127.9.225.1@KRBTEST.COM (short username kudu)
I20260809 00:57:58.643684 8894 webserver.cc:460] Webserver started at http://127.9.225.1:37241/ using document root <none> and password file <none>
I20260809 00:57:58.644552 8894 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:58.644724 8894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:58.645049 8894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:58.648554 8894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "f8e64f9f4adb4d73986fd8b17cf74cea"
format_stamp: "Formatted at 2026-08-09 00:57:58 on dist-test-slave-j2vl"
server_key: "3bffd88c1490f0fbbe601b67f8cb4d5d"
server_key_iv: "852060ba6b7bfb8aa07dc2e253f16b5f"
server_key_version: "encryptionkey@0"
I20260809 00:57:58.649505 8894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "f8e64f9f4adb4d73986fd8b17cf74cea"
format_stamp: "Formatted at 2026-08-09 00:57:58 on dist-test-slave-j2vl"
server_key: "3bffd88c1490f0fbbe601b67f8cb4d5d"
server_key_iv: "852060ba6b7bfb8aa07dc2e253f16b5f"
server_key_version: "encryptionkey@0"
I20260809 00:57:58.655686 8894 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260809 00:57:58.660104 8916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:58.661939 8894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:57:58.662132 8894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "f8e64f9f4adb4d73986fd8b17cf74cea"
format_stamp: "Formatted at 2026-08-09 00:57:58 on dist-test-slave-j2vl"
server_key: "3bffd88c1490f0fbbe601b67f8cb4d5d"
server_key_iv: "852060ba6b7bfb8aa07dc2e253f16b5f"
server_key_version: "encryptionkey@0"
I20260809 00:57:58.662358 8894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:58.683027 8894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:58.688046 8894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:58.688355 8894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:58.689540 8894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:58.691449 8894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:58.691541 8894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:58.691661 8894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:58.691707 8894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:58.730113 8894 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:38005
I20260809 00:57:58.730180 9121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:38005 every 8 connection(s)
I20260809 00:57:58.731645 8894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:57:58.732167 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 8894
I20260809 00:57:58.732376 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:57:58.733057 10116 external_mini_cluster.cc:1428] Setting key 11d5f2a63ebadad1944a314dd2e16777
Aug 09 00:57:58 dist-test-slave-j2vl krb5kdc[8707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237078, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.1@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:57:58.756124 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:58.735124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:50125 (local address 127.9.225.62:42371)
0809 00:57:58.735636 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:58.735643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:58.737129 (+ 1486us) server_negotiation.cc:407] Connection header received
0809 00:57:58.739274 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:58.739280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:58.739394 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:58.739529 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:58.741266 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:58.742084 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:58.743955 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:58.744542 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:58.744685 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:58.748787 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:58.748793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:58.748796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:58.748838 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:58.752014 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:58.753064 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:58.753072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:58.753076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:58.753250 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:58.753844 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:58.753858 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:58.753862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:58.754205 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:58.754409 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:58.755387 (+ 978us) server_negotiation.cc:299] Negotiation successful
0809 00:57:58.755695 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260809 00:57:58.757567 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:58.735980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:42371 (local address 127.9.225.1:50125)
0809 00:57:58.736870 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:58.736994 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:58.738957 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:58.739808 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:58.739817 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:58.740433 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:58.741070 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:58.741089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:58.742278 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:58.742282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:58.743784 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:58.743810 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:58.744806 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:58.744814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:58.745119 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:58.745967 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:57:58.746000 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:58.748484 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:58.752255 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:58.752268 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:58.752300 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:58.752888 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:58.753437 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:58.753443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:58.753450 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:58.753692 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:58.754389 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:58.754400 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:58.754892 (+ 492us) client_negotiation.cc:769] Sending connection context
0809 00:57:58.755231 (+ 339us) client_negotiation.cc:240] Negotiation successful
0809 00:57:58.755923 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":358,"threads_started":1}
I20260809 00:57:58.759716 9123 heartbeater.cc:347] Connected to a master server at 127.9.225.62:42371
I20260809 00:57:58.760167 9123 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:58.761137 9123 heartbeater.cc:511] Master 127.9.225.62:42371 requested a full tablet report, sending...
I20260809 00:57:58.764292 8782 ts_manager.cc:194] Registered new tserver with Master: f8e64f9f4adb4d73986fd8b17cf74cea (127.9.225.1:38005)
I20260809 00:57:58.766961 8782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.1@KRBTEST.COM'} at 127.9.225.1:50125
WARNING: no policy specified for kudu/127.9.225.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
Entry for principal kudu/127.9.225.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab.
I20260809 00:57:58.816617 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42371
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:57:59.212783 9130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:59.213128 9130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:59.213292 9130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:59.221041 9130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:59.221191 9130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:59.221431 9130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:57:59.232579 9130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.2.keytab
--principal=kudu/127.9.225.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:59.234385 9130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:59.236040 9130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:59.251783 9138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:59.251652 9137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:59.251834 9144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:59.252149 9130 server_base.cc:1034] running on GCE node
I20260809 00:57:59.253315 9130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:59.254575 9130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:59.255832 9130 hybrid_clock.cc:648] HybridClock initialized: now 1786237079255778 us; error 49 us; skew 500 ppm
I20260809 00:57:59.256270 9130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:59 dist-test-slave-j2vl krb5kdc[8707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237079, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:59.260586 9130 init.cc:377] Logged in from keytab as kudu/127.9.225.2@KRBTEST.COM (short username kudu)
I20260809 00:57:59.262611 9130 webserver.cc:460] Webserver started at http://127.9.225.2:33063/ using document root <none> and password file <none>
I20260809 00:57:59.263486 9130 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:59.263643 9130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:59.264034 9130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:59.267817 9130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "c95b59a70a474f2fa4823d753031f5e2"
format_stamp: "Formatted at 2026-08-09 00:57:59 on dist-test-slave-j2vl"
server_key: "623e42cd029df46d82fae510715bc2f1"
server_key_iv: "85e18721232de5ccbe50abc5e28c3800"
server_key_version: "encryptionkey@0"
I20260809 00:57:59.268807 9130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "c95b59a70a474f2fa4823d753031f5e2"
format_stamp: "Formatted at 2026-08-09 00:57:59 on dist-test-slave-j2vl"
server_key: "623e42cd029df46d82fae510715bc2f1"
server_key_iv: "85e18721232de5ccbe50abc5e28c3800"
server_key_version: "encryptionkey@0"
I20260809 00:57:59.275372 9130 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260809 00:57:59.279649 9155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:59.281594 9130 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:57:59.281836 9130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "c95b59a70a474f2fa4823d753031f5e2"
format_stamp: "Formatted at 2026-08-09 00:57:59 on dist-test-slave-j2vl"
server_key: "623e42cd029df46d82fae510715bc2f1"
server_key_iv: "85e18721232de5ccbe50abc5e28c3800"
server_key_version: "encryptionkey@0"
I20260809 00:57:59.282121 9130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:59.313906 9130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:59.319185 9130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:59.319589 9130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:59.320851 9130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:59.322654 9130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:59.322742 9130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:59.322933 9130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:59.323004 9130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:59.362486 9130 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:37719
I20260809 00:57:59.362541 9347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:37719 every 8 connection(s)
I20260809 00:57:59.364040 9130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:57:59.365903 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 9130
I20260809 00:57:59.366128 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:57:59.367146 10116 external_mini_cluster.cc:1428] Setting key 481468e728b7de47a8d0cf3a5b71e8db
Aug 09 00:57:59 dist-test-slave-j2vl krb5kdc[8707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237079, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.2@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:57:59.389827 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:59.367868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:43571 (local address 127.9.225.62:42371)
0809 00:57:59.368472 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0809 00:57:59.368477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:57:59.370238 (+ 1761us) server_negotiation.cc:407] Connection header received
0809 00:57:59.372232 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:57:59.372238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:57:59.372355 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:57:59.372484 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:57:59.374328 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:59.375133 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:59.376825 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:57:59.377295 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:57:59.377491 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:59.381797 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:57:59.381803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:57:59.381807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:57:59.381848 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:57:59.385433 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:59.386743 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:59.386752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:59.386757 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:59.386940 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:57:59.387460 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:57:59.387468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:57:59.387474 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:57:59.387949 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:57:59.388115 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 00:57:59.388946 (+ 831us) server_negotiation.cc:299] Negotiation successful
0809 00:57:59.389269 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":293,"threads_started":1}
I20260809 00:57:59.390733 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:57:59.368644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:42371 (local address 127.9.225.2:43571)
0809 00:57:59.370028 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0809 00:57:59.370098 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0809 00:57:59.371927 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:57:59.372777 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:57:59.372786 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:57:59.373478 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:57:59.374129 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:59.374148 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:59.375332 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:59.375336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:59.376649 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:57:59.376662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:57:59.377608 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:57:59.377616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:57:59.377886 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:57:59.378699 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:57:59.378726 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:57:59.381459 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:57:59.385723 (+ 4264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:59.385733 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:59.385752 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:59.386510 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:59.387073 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:57:59.387077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:57:59.387080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:57:59.387326 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:57:59.388076 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:57:59.388083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:57:59.388511 (+ 428us) client_negotiation.cc:769] Sending connection context
0809 00:57:59.388787 (+ 276us) client_negotiation.cc:240] Negotiation successful
0809 00:57:59.389264 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"thread_start_us":738,"threads_started":1}
I20260809 00:57:59.392648 9349 heartbeater.cc:347] Connected to a master server at 127.9.225.62:42371
I20260809 00:57:59.393075 9349 heartbeater.cc:464] Registering TS with master...
I20260809 00:57:59.394193 9349 heartbeater.cc:511] Master 127.9.225.62:42371 requested a full tablet report, sending...
I20260809 00:57:59.396334 8782 ts_manager.cc:194] Registered new tserver with Master: c95b59a70a474f2fa4823d753031f5e2 (127.9.225.2:37719)
I20260809 00:57:59.397332 8782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.2@KRBTEST.COM'} at 127.9.225.2:43571
WARNING: no policy specified for kudu/127.9.225.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
Entry for principal kudu/127.9.225.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab.
I20260809 00:57:59.454756 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42371
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:57:59.771348 9123 heartbeater.cc:503] Master 127.9.225.62:42371 was elected leader, sending a full tablet report...
W20260809 00:57:59.857987 9357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:57:59.858387 9357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:57:59.858498 9357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:57:59.866989 9357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:57:59.867137 9357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:57:59.867260 9357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:57:59.878690 9357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.3.keytab
--principal=kudu/127.9.225.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:42371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:57:59.881307 9357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:57:59.883749 9357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:57:59.903821 9364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:59.903798 9368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:57:59.904017 9372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:57:59.905009 9357 server_base.cc:1034] running on GCE node
I20260809 00:57:59.905838 9357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:57:59.906986 9357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:57:59.908241 9357 hybrid_clock.cc:648] HybridClock initialized: now 1786237079908184 us; error 55 us; skew 500 ppm
I20260809 00:57:59.908612 9357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:57:59 dist-test-slave-j2vl krb5kdc[8707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237079, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:57:59.912324 9357 init.cc:377] Logged in from keytab as kudu/127.9.225.3@KRBTEST.COM (short username kudu)
I20260809 00:57:59.914222 9357 webserver.cc:460] Webserver started at http://127.9.225.3:34069/ using document root <none> and password file <none>
I20260809 00:57:59.915056 9357 fs_manager.cc:359] Metadata directory not provided
I20260809 00:57:59.915207 9357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:57:59.915530 9357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:57:59.918963 9357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f"
format_stamp: "Formatted at 2026-08-09 00:57:59 on dist-test-slave-j2vl"
server_key: "7c28b21d607c161468053479c8a5658d"
server_key_iv: "64bd5d7f53a187944ef5356baba812a4"
server_key_version: "encryptionkey@0"
I20260809 00:57:59.919833 9357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f"
format_stamp: "Formatted at 2026-08-09 00:57:59 on dist-test-slave-j2vl"
server_key: "7c28b21d607c161468053479c8a5658d"
server_key_iv: "64bd5d7f53a187944ef5356baba812a4"
server_key_version: "encryptionkey@0"
I20260809 00:57:59.926108 9357 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 00:57:59.930368 9386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:57:59.932102 9357 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:57:59.932351 9357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f"
format_stamp: "Formatted at 2026-08-09 00:57:59 on dist-test-slave-j2vl"
server_key: "7c28b21d607c161468053479c8a5658d"
server_key_iv: "64bd5d7f53a187944ef5356baba812a4"
server_key_version: "encryptionkey@0"
I20260809 00:57:59.932585 9357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:57:59.987850 9357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:57:59.993413 9357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:57:59.993794 9357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:57:59.995087 9357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:57:59.997273 9357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:57:59.997413 9357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 00:57:59.997541 9357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:57:59.997602 9357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:00.037029 9357 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:41287
I20260809 00:58:00.037108 9587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:41287 every 8 connection(s)
I20260809 00:58:00.038772 9357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:58:00.048452 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 9357
I20260809 00:58:00.048693 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:58:00.049216 10116 external_mini_cluster.cc:1428] Setting key 560298374a563c3e422f1e53e28f4fa7
Aug 09 00:58:00 dist-test-slave-j2vl krb5kdc[8707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237079, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.3@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:58:00.062939 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.041830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:52783 (local address 127.9.225.62:42371)
0809 00:58:00.042313 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.042318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.043521 (+ 1203us) server_negotiation.cc:407] Connection header received
0809 00:58:00.045856 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.045862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.045987 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.046130 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:00.047986 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.048917 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.050965 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.051443 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.051600 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.056081 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:00.056088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:00.056091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:00.056132 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:00.059228 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:00.060176 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:00.060188 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:00.060191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:00.060332 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:00.060835 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:00.060843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:00.060851 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:00.061164 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:00.061355 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.062218 (+ 863us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.062506 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":233,"threads_started":1}
I20260809 00:58:00.064327 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.042505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:42371 (local address 127.9.225.3:52783)
0809 00:58:00.043312 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:00.043358 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:00.045492 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:00.046421 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:00.046431 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:00.047114 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:00.047757 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.047778 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.049106 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.049111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.050777 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.050793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.051757 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.051767 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.052025 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.052777 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:00.052819 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:00.055735 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:00.059429 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:00.059443 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:00.059471 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:00.060011 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:00.060468 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:00.060473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:00.060477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:00.060701 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:00.061331 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:00.061362 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:00.061793 (+ 431us) client_negotiation.cc:769] Sending connection context
0809 00:58:00.062107 (+ 314us) client_negotiation.cc:240] Negotiation successful
0809 00:58:00.062779 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":328,"threads_started":1}
I20260809 00:58:00.066004 9590 heartbeater.cc:347] Connected to a master server at 127.9.225.62:42371
I20260809 00:58:00.066438 9590 heartbeater.cc:464] Registering TS with master...
I20260809 00:58:00.067448 9590 heartbeater.cc:511] Master 127.9.225.62:42371 requested a full tablet report, sending...
I20260809 00:58:00.069448 8781 ts_manager.cc:194] Registered new tserver with Master: e5b30c89dc2f4f998fe7e7f0c7f75d8f (127.9.225.3:41287)
I20260809 00:58:00.070494 8781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.225.3@KRBTEST.COM'} at 127.9.225.3:52783
I20260809 00:58:00.079012 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:58:00.100044 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.084208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.9.225.62:42371)
0809 00:58:00.084508 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.084515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.084845 (+ 330us) server_negotiation.cc:407] Connection header received
0809 00:58:00.085086 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.085090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.085165 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.085282 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:00.086984 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.087761 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.089207 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.089739 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.089903 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.092779 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:00.092786 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:00.092789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:00.092834 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:00.096296 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:00.097326 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:00.097331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:00.097333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:00.097581 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:00.098120 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:00.098125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:00.098128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:00.098700 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:00.098939 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.099401 (+ 462us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.099634 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260809 00:58:00.105495 8781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:58:00.113597 8781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:58:00.144503 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.130539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.9.225.1:38005)
0809 00:58:00.134064 (+ 3525us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.134076 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.134108 (+ 32us) server_negotiation.cc:407] Connection header received
0809 00:58:00.134216 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.134223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.134498 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.135349 (+ 851us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:58:00.136710 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.138507 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.141668 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.142895 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.142985 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.143287 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.143749 (+ 462us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.144122 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3312,"thread_start_us":3136,"threads_started":1}
I20260809 00:58:00.144991 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.130242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38005 (local address 127.0.0.1:46846)
0809 00:58:00.131111 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:00.131158 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:00.131390 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:00.134825 (+ 3435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:00.134834 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:00.134879 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:58:00.136428 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.136446 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.138717 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.138726 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.141432 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.141446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.143287 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.143294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.143490 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.143527 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 00:58:00.143697 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 00:58:00.144062 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":472,"thread_start_us":377,"threads_started":1}
I20260809 00:58:00.145958 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.134251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:41287 (local address 127.0.0.1:42732)
0809 00:58:00.136800 (+ 2549us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:00.136816 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:00.137049 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:00.137820 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:00.137825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:00.137854 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:58:00.138270 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.138282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.139964 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.139970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.142275 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.142293 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.144961 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.144971 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.145191 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.145228 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 00:58:00.145336 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 00:58:00.145515 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2374,"mutex_wait_us":45,"thread_start_us":1638,"threads_started":1}
I20260809 00:58:00.146258 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.134781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.9.225.3:41287)
0809 00:58:00.136670 (+ 1889us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.136682 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.136985 (+ 303us) server_negotiation.cc:407] Connection header received
0809 00:58:00.137258 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.137268 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.137630 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.137832 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:58:00.138462 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.139809 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.142504 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.144650 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.144885 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.145198 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.145480 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.145695 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1653,"thread_start_us":1428,"threads_started":1}
I20260809 00:58:00.148948 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.131378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.9.225.2:37719)
0809 00:58:00.133282 (+ 1904us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.133297 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.133369 (+ 72us) server_negotiation.cc:407] Connection header received
0809 00:58:00.133988 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.134002 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.134333 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.136277 (+ 1944us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:58:00.139349 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.141217 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.144046 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.147705 (+ 3659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.147882 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.148118 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.148426 (+ 308us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.148607 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1669,"thread_start_us":1480,"threads_started":1}
I20260809 00:58:00.148998 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.130666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:37719 (local address 127.0.0.1:35604)
0809 00:58:00.132369 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:00.132411 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:00.132590 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:00.134734 (+ 2144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:00.134745 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:00.134787 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:58:00.135514 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.135537 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.141445 (+ 5908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.141452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.143825 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.143842 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.147961 (+ 4119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.147970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.148183 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.148222 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 00:58:00.148345 (+ 123us) client_negotiation.cc:240] Negotiation successful
0809 00:58:00.148513 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1511,"mutex_wait_us":83,"thread_start_us":1315,"threads_started":1}
I20260809 00:58:00.150419 9006 tablet_service.cc:1467] Processing CreateTablet for tablet 6dd6d95b947d46bfa74727e3688ebe71 (DEFAULT_TABLE table=test-table [id=202a1406a91f491a8ab7efd6bebe47e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:58:00.150521 9466 tablet_service.cc:1467] Processing CreateTablet for tablet 6dd6d95b947d46bfa74727e3688ebe71 (DEFAULT_TABLE table=test-table [id=202a1406a91f491a8ab7efd6bebe47e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:58:00.152216 9006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dd6d95b947d46bfa74727e3688ebe71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:00.152208 9466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dd6d95b947d46bfa74727e3688ebe71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:00.153318 9240 tablet_service.cc:1467] Processing CreateTablet for tablet 6dd6d95b947d46bfa74727e3688ebe71 (DEFAULT_TABLE table=test-table [id=202a1406a91f491a8ab7efd6bebe47e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:58:00.155095 9240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dd6d95b947d46bfa74727e3688ebe71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:00.168040 9623 tablet_bootstrap.cc:492] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea: Bootstrap starting.
I20260809 00:58:00.172297 9623 tablet_bootstrap.cc:654] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:00.173484 9629 tablet_bootstrap.cc:492] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f: Bootstrap starting.
I20260809 00:58:00.173933 9623 log.cc:824] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:00.178004 9629 tablet_bootstrap.cc:654] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:00.178584 9623 tablet_bootstrap.cc:492] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea: No bootstrap required, opened a new log
I20260809 00:58:00.177392 9633 tablet_bootstrap.cc:492] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2: Bootstrap starting.
I20260809 00:58:00.178936 9623 ts_tablet_manager.cc:1397] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260809 00:58:00.179834 9629 log.cc:824] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:00.183105 9633 tablet_bootstrap.cc:654] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:00.184552 9633 log.cc:824] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:00.184702 9629 tablet_bootstrap.cc:492] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f: No bootstrap required, opened a new log
I20260809 00:58:00.185038 9629 ts_tablet_manager.cc:1397] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260809 00:58:00.188380 9633 tablet_bootstrap.cc:492] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2: No bootstrap required, opened a new log
I20260809 00:58:00.188719 9633 ts_tablet_manager.cc:1397] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260809 00:58:00.188849 9623 raft_consensus.cc:348] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38005 } } peers { permanent_uuid: "c95b59a70a474f2fa4823d753031f5e2" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37719 } } peers { permanent_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 41287 } }
I20260809 00:58:00.189441 9623 raft_consensus.cc:374] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:00.189561 9623 raft_consensus.cc:729] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e64f9f4adb4d73986fd8b17cf74cea, State: Initialized, Role: FOLLOWER
I20260809 00:58:00.190303 9623 consensus_queue.cc:260] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38005 } } peers { permanent_uuid: "c95b59a70a474f2fa4823d753031f5e2" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37719 } } peers { permanent_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 41287 } }
I20260809 00:58:00.192420 9623 ts_tablet_manager.cc:1428] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260809 00:58:00.195225 9629 raft_consensus.cc:348] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38005 } } peers { permanent_uuid: "c95b59a70a474f2fa4823d753031f5e2" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37719 } } peers { permanent_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 41287 } }
I20260809 00:58:00.195715 9629 raft_consensus.cc:374] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:00.195837 9629 raft_consensus.cc:729] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5b30c89dc2f4f998fe7e7f0c7f75d8f, State: Initialized, Role: FOLLOWER
I20260809 00:58:00.196650 9629 consensus_queue.cc:260] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38005 } } peers { permanent_uuid: "c95b59a70a474f2fa4823d753031f5e2" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37719 } } peers { permanent_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 41287 } }
I20260809 00:58:00.198033 9590 heartbeater.cc:503] Master 127.9.225.62:42371 was elected leader, sending a full tablet report...
I20260809 00:58:00.199677 9629 ts_tablet_manager.cc:1428] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260809 00:58:00.201891 9633 raft_consensus.cc:348] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38005 } } peers { permanent_uuid: "c95b59a70a474f2fa4823d753031f5e2" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37719 } } peers { permanent_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 41287 } }
I20260809 00:58:00.202339 9633 raft_consensus.cc:374] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:00.202479 9633 raft_consensus.cc:729] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95b59a70a474f2fa4823d753031f5e2, State: Initialized, Role: FOLLOWER
I20260809 00:58:00.203184 9633 consensus_queue.cc:260] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38005 } } peers { permanent_uuid: "c95b59a70a474f2fa4823d753031f5e2" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37719 } } peers { permanent_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 41287 } }
I20260809 00:58:00.204726 9349 heartbeater.cc:503] Master 127.9.225.62:42371 was elected leader, sending a full tablet report...
I20260809 00:58:00.205588 9633 ts_tablet_manager.cc:1428] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
W20260809 00:58:00.234999 9124 tablet.cc:2369] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:58:00.290735 9591 tablet.cc:2369] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:58:00.366761 9350 tablet.cc:2369] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:58:00.376020 9643 raft_consensus.cc:482] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:58:00.376417 9643 raft_consensus.cc:504] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38005 } } peers { permanent_uuid: "c95b59a70a474f2fa4823d753031f5e2" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37719 } } peers { permanent_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 41287 } }
I20260809 00:58:00.378615 9643 leader_election.cc:288] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e64f9f4adb4d73986fd8b17cf74cea (127.9.225.1:38005), e5b30c89dc2f4f998fe7e7f0c7f75d8f (127.9.225.3:41287)
I20260809 00:58:00.379802 9637 raft_consensus.cc:482] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:58:00.380316 9637 raft_consensus.cc:504] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38005 } } peers { permanent_uuid: "c95b59a70a474f2fa4823d753031f5e2" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37719 } } peers { permanent_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 41287 } }
I20260809 00:58:00.389158 9637 leader_election.cc:288] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95b59a70a474f2fa4823d753031f5e2 (127.9.225.2:37719), e5b30c89dc2f4f998fe7e7f0c7f75d8f (127.9.225.3:41287)
I20260809 00:58:00.389905 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.379255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:59845 (local address 127.9.225.1:38005)
0809 00:58:00.379630 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.379638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.380045 (+ 407us) server_negotiation.cc:407] Connection header received
0809 00:58:00.380405 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.380411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.380511 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.380587 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:58:00.381471 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.382541 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.385864 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.387821 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.388014 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.388167 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.389119 (+ 952us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.389379 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260809 00:58:00.391458 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.378927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:38005 (local address 127.9.225.2:59845)
0809 00:58:00.379928 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:00.379952 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:00.380178 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:00.380837 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:00.380841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:00.380868 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:58:00.381270 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.381285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.382690 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.382696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.385543 (+ 2847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.385572 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.388492 (+ 2920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.388502 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.388829 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.388889 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 00:58:00.390296 (+ 1407us) client_negotiation.cc:240] Negotiation successful
0809 00:58:00.390699 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":1203,"threads_started":1}
I20260809 00:58:00.393121 9040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd6d95b947d46bfa74727e3688ebe71" candidate_uuid: "c95b59a70a474f2fa4823d753031f5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" is_pre_election: true
I20260809 00:58:00.393829 9040 raft_consensus.cc:2454] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c95b59a70a474f2fa4823d753031f5e2 in term 0.
I20260809 00:58:00.395054 9163 leader_election.cc:302] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95b59a70a474f2fa4823d753031f5e2, f8e64f9f4adb4d73986fd8b17cf74cea; no voters:
I20260809 00:58:00.395627 9643 raft_consensus.cc:2781] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:58:00.395823 9643 raft_consensus.cc:482] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:58:00.395973 9643 raft_consensus.cc:3037] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:00.397051 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.379119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:53389 (local address 127.9.225.3:41287)
0809 00:58:00.379340 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.379347 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.380582 (+ 1235us) server_negotiation.cc:407] Connection header received
0809 00:58:00.380822 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.380830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.380915 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.381022 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:58:00.381852 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.383024 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.393237 (+ 10213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.395543 (+ 2306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.395717 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.395870 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.396578 (+ 708us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.396727 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 00:58:00.397295 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.378986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:41287 (local address 127.9.225.2:53389)
0809 00:58:00.380453 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:00.380469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:00.380633 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:00.381094 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:00.381099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:00.381129 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:58:00.381651 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.381664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.383214 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.383221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.392131 (+ 8910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.392165 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.396079 (+ 3914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.396088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.396333 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.396377 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 00:58:00.396470 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 00:58:00.396720 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"mutex_wait_us":80,"thread_start_us":552,"threads_started":1}
I20260809 00:58:00.398583 9507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd6d95b947d46bfa74727e3688ebe71" candidate_uuid: "c95b59a70a474f2fa4823d753031f5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" is_pre_election: true
I20260809 00:58:00.399350 9507 raft_consensus.cc:2454] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c95b59a70a474f2fa4823d753031f5e2 in term 0.
I20260809 00:58:00.399887 9643 raft_consensus.cc:504] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38005 } } peers { permanent_uuid: "c95b59a70a474f2fa4823d753031f5e2" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37719 } } peers { permanent_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 41287 } }
I20260809 00:58:00.401614 9643 leader_election.cc:288] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [CANDIDATE]: Term 1 election: Requested vote from peers f8e64f9f4adb4d73986fd8b17cf74cea (127.9.225.1:38005), e5b30c89dc2f4f998fe7e7f0c7f75d8f (127.9.225.3:41287)
I20260809 00:58:00.401769 9040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd6d95b947d46bfa74727e3688ebe71" candidate_uuid: "c95b59a70a474f2fa4823d753031f5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea"
I20260809 00:58:00.402072 9040 raft_consensus.cc:3037] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:00.402866 9507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd6d95b947d46bfa74727e3688ebe71" candidate_uuid: "c95b59a70a474f2fa4823d753031f5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f"
I20260809 00:58:00.403122 9507 raft_consensus.cc:3037] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:00.403494 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.383355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:41287 (local address 127.9.225.1:42165)
0809 00:58:00.392510 (+ 9155us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:00.392533 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:00.392703 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:00.394286 (+ 1583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:00.394293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:00.394328 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:58:00.395009 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.395031 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.397367 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.397373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.399983 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.400004 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.402529 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.402541 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.402724 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.402763 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 00:58:00.402873 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 00:58:00.403044 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8977,"thread_start_us":12032,"threads_started":1}
I20260809 00:58:00.404481 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.383807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:42165 (local address 127.9.225.3:41287)
0809 00:58:00.393851 (+ 10044us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.393864 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.393892 (+ 28us) server_negotiation.cc:407] Connection header received
0809 00:58:00.393968 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.393975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.394065 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.395198 (+ 1133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:58:00.395628 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.397064 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.400361 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.402147 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.402395 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.402533 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.403868 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.404052 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9869,"thread_start_us":13488,"threads_started":1}
I20260809 00:58:00.405488 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.383373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:37719 (local address 127.9.225.1:51311)
0809 00:58:00.395744 (+ 12371us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:00.395767 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:00.396127 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:00.397908 (+ 1781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:00.397917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:00.397957 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:58:00.398592 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.398614 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.400192 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.400197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.402364 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:00.402389 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.404412 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.404429 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:00.404623 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.404788 (+ 165us) client_negotiation.cc:769] Sending connection context
0809 00:58:00.404894 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 00:58:00.405051 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11875,"mutex_wait_us":7,"thread_start_us":9426,"threads_started":1}
I20260809 00:58:00.405602 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.383886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:51311 (local address 127.9.225.2:37719)
0809 00:58:00.384097 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.384103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.395934 (+ 11831us) server_negotiation.cc:407] Connection header received
0809 00:58:00.396298 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.396304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.396390 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.396457 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:58:00.398880 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.399983 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.402624 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.403909 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.404036 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.404146 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.405000 (+ 854us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.405176 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":72}
I20260809 00:58:00.406054 9040 raft_consensus.cc:2454] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95b59a70a474f2fa4823d753031f5e2 in term 1.
I20260809 00:58:00.406576 9163 leader_election.cc:302] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c95b59a70a474f2fa4823d753031f5e2, f8e64f9f4adb4d73986fd8b17cf74cea; no voters:
I20260809 00:58:00.406708 9280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd6d95b947d46bfa74727e3688ebe71" candidate_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95b59a70a474f2fa4823d753031f5e2" is_pre_election: true
I20260809 00:58:00.406922 9643 raft_consensus.cc:2781] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:00.407112 9504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dd6d95b947d46bfa74727e3688ebe71" candidate_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" is_pre_election: true
I20260809 00:58:00.407653 9643 raft_consensus.cc:686] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 1 LEADER]: Becoming Leader. State: Replica: c95b59a70a474f2fa4823d753031f5e2, State: Running, Role: LEADER
I20260809 00:58:00.408299 9643 consensus_queue.cc:237] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38005 } } peers { permanent_uuid: "c95b59a70a474f2fa4823d753031f5e2" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37719 } } peers { permanent_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 41287 } }
I20260809 00:58:00.409565 9507 raft_consensus.cc:2454] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c95b59a70a474f2fa4823d753031f5e2 in term 1.
I20260809 00:58:00.413996 8920 leader_election.cc:302] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e64f9f4adb4d73986fd8b17cf74cea; no voters: c95b59a70a474f2fa4823d753031f5e2, e5b30c89dc2f4f998fe7e7f0c7f75d8f
I20260809 00:58:00.414559 9637 raft_consensus.cc:2726] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 00:58:00.414937 8782 catalog_manager.cc:5495] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c95b59a70a474f2fa4823d753031f5e2 (127.9.225.2). New cstate: current_term: 1 leader_uuid: "c95b59a70a474f2fa4823d753031f5e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c95b59a70a474f2fa4823d753031f5e2" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 37719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 41287 } health_report { overall_health: UNKNOWN } } }
I20260809 00:58:00.450892 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.444052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.9.225.2:37719)
0809 00:58:00.444317 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.444323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.444396 (+ 73us) server_negotiation.cc:407] Connection header received
0809 00:58:00.444696 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.444701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.444794 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.444967 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:58:00.445474 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.446250 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.447715 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.448255 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.448356 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.448991 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:58:00.449711 (+ 720us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:58:00.449846 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.450254 (+ 408us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.450472 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 00:58:00.461177 9507 raft_consensus.cc:1261] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f [term 1 FOLLOWER]: Refusing update from remote peer c95b59a70a474f2fa4823d753031f5e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:58:00.461354 9040 raft_consensus.cc:1261] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [term 1 FOLLOWER]: Refusing update from remote peer c95b59a70a474f2fa4823d753031f5e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:58:00.462579 9651 consensus_queue.cc:1035] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e64f9f4adb4d73986fd8b17cf74cea" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 00:58:00.463209 9643 consensus_queue.cc:1035] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b30c89dc2f4f998fe7e7f0c7f75d8f" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 41287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:58:00.482311 9656 mvcc.cc:205] Tried to move back new op lower bound from 7316427081557245952 to 7316427081358028800. Current Snapshot: MvccSnapshot[applied={T|T < 7316427081557245952}]
I20260809 00:58:00.504585 8782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:58:00.504901 8782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:58:00.509223 8782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1557df8496441f4b38ec3781f1cfecd: Sending DeleteTablet for 3 replicas of tablet 6dd6d95b947d46bfa74727e3688ebe71
I20260809 00:58:00.510753 9240 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dd6d95b947d46bfa74727e3688ebe71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:58:00 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:35604
I20260809 00:58:00.510744 9006 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dd6d95b947d46bfa74727e3688ebe71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:58:00 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:46846
I20260809 00:58:00.511073 9466 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dd6d95b947d46bfa74727e3688ebe71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:58:00 UTC) from {username='kudu', principal='kudu/127.9.225.62@KRBTEST.COM'} at 127.0.0.1:42732
I20260809 00:58:00.516124 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 8894
I20260809 00:58:00.528960 9675 tablet_replica.cc:330] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea: stopping tablet replica
I20260809 00:58:00.529829 9675 raft_consensus.cc:2229] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:58:00.530596 9675 raft_consensus.cc:2258] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:58:00.533548 9675 ts_tablet_manager.cc:1905] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:58:00.533877 9676 tablet_replica.cc:330] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f: stopping tablet replica
I20260809 00:58:00.534428 9676 raft_consensus.cc:2229] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:58:00.535071 9676 raft_consensus.cc:2258] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:58:00.534560 9673 tablet_replica.cc:330] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2: stopping tablet replica
I20260809 00:58:00.536877 9673 raft_consensus.cc:2229] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:58:00.537775 9676 ts_tablet_manager.cc:1905] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 00:58:00 dist-test-slave-j2vl krb5kdc[8707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.1@KRBTEST.COM
I20260809 00:58:00.540460 9673 raft_consensus.cc:2258] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:58:00.543747 9673 ts_tablet_manager.cc:1905] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:58:00.545591 9676 ts_tablet_manager.cc:1918] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:58:00.545814 9676 log.cc:1196] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/6dd6d95b947d46bfa74727e3688ebe71
I20260809 00:58:00.546420 9676 ts_tablet_manager.cc:1939] T 6dd6d95b947d46bfa74727e3688ebe71 P e5b30c89dc2f4f998fe7e7f0c7f75d8f: Deleting consensus metadata
I20260809 00:58:00.548743 8761 catalog_manager.cc:4841] TS e5b30c89dc2f4f998fe7e7f0c7f75d8f (127.9.225.3:41287): tablet 6dd6d95b947d46bfa74727e3688ebe71 (table test-table [id=202a1406a91f491a8ab7efd6bebe47e2]) successfully deleted
I20260809 00:58:00.551805 9675 ts_tablet_manager.cc:1918] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:58:00.552054 9675 log.cc:1196] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-0/wal/wals/6dd6d95b947d46bfa74727e3688ebe71
I20260809 00:58:00.552693 9675 ts_tablet_manager.cc:1939] T 6dd6d95b947d46bfa74727e3688ebe71 P f8e64f9f4adb4d73986fd8b17cf74cea: Deleting consensus metadata
I20260809 00:58:00.553505 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.517677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.9.225.1:38005)
0809 00:58:00.517949 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.517958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.522881 (+ 4923us) server_negotiation.cc:407] Connection header received
0809 00:58:00.524009 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.524020 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.524149 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.524250 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:00.529236 (+ 4986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.530497 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.533759 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.534467 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.534594 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.541848 (+ 7254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:00.541917 (+ 69us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:00.541939 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:00.542001 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:00.546599 (+ 4598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:00.549616 (+ 3017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:00.549631 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:00.549635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:00.549892 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:00.550668 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:00.550674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:00.550678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:00.551253 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:00.551747 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.552646 (+ 899us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.552974 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 00:58:00.554044 9673 ts_tablet_manager.cc:1918] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:58:00.554322 9673 log.cc:1196] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/6dd6d95b947d46bfa74727e3688ebe71
I20260809 00:58:00.555147 8765 catalog_manager.cc:4841] TS f8e64f9f4adb4d73986fd8b17cf74cea (127.9.225.1:38005): tablet 6dd6d95b947d46bfa74727e3688ebe71 (table test-table [id=202a1406a91f491a8ab7efd6bebe47e2]) successfully deleted
I20260809 00:58:00.555256 9113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:00.556881 9673 ts_tablet_manager.cc:1939] T 6dd6d95b947d46bfa74727e3688ebe71 P c95b59a70a474f2fa4823d753031f5e2: Deleting consensus metadata
I20260809 00:58:00.560027 8763 catalog_manager.cc:4841] TS c95b59a70a474f2fa4823d753031f5e2 (127.9.225.2:37719): tablet 6dd6d95b947d46bfa74727e3688ebe71 (table test-table [id=202a1406a91f491a8ab7efd6bebe47e2]) successfully deleted
I20260809 00:58:00.654866 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 8894
I20260809 00:58:00.689942 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 9130
Aug 09 00:58:00 dist-test-slave-j2vl krb5kdc[8707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.2@KRBTEST.COM
I20260809 00:58:00.711651 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.690936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.9.225.2:37719)
0809 00:58:00.691147 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.691155 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.691178 (+ 23us) server_negotiation.cc:407] Connection header received
0809 00:58:00.691345 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.691351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.691441 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.691560 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:00.692975 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.694212 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.695942 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.696631 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.696775 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.702866 (+ 6091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:00.702893 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:00.702922 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:00.703006 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:00.707713 (+ 4707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:00.708726 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:00.708738 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:00.708742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:00.708943 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:00.709628 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:00.709634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:00.709637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:00.710279 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:00.710644 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.710921 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.711201 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 00:58:00.712518 9343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:00.844862 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 9130
I20260809 00:58:00.874399 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 9357
Aug 09 00:58:00 dist-test-slave-j2vl krb5kdc[8707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.3@KRBTEST.COM
I20260809 00:58:00.892642 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:00.875403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.9.225.3:41287)
0809 00:58:00.875617 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:00.875625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:00.875648 (+ 23us) server_negotiation.cc:407] Connection header received
0809 00:58:00.875869 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:00.875873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:00.875949 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:00.876068 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:00.877559 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.878292 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.879659 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:00.880089 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:00.880234 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:00.885674 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:00.885694 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:00.885714 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:00.885765 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:00.888936 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:00.890183 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:00.890193 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:00.890196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:00.890377 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:00.890852 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:00.890856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:00.890859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:00.891275 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:00.891655 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:00.891997 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 00:58:00.892271 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 00:58:00.893517 9583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:00.989101 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 9357
I20260809 00:58:01.018927 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 8729
I20260809 00:58:01.020226 8839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:01.139103 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 8729
2026-08-09T00:58:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5460 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T00:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:01Z Disabled control of system clock
I20260809 00:58:01.255754 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:36169
--webserver_interface=127.9.225.62
--webserver_port=0
--builtin_ntp_servers=127.9.225.20:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:36169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 00:58:01.651005 9689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:01.651381 9689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:01.651471 9689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:01.659898 9689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:01.660132 9689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:01.660181 9689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:01.660261 9689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:01.660311 9689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:01.671928 9689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:36169
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:36169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:01.674088 9689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:01.676379 9689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:01.689738 9696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:01.689754 9698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:01.689682 9702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:01.692174 9689 server_base.cc:1034] running on GCE node
I20260809 00:58:01.693300 9689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:01.695252 9689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:01.696540 9689 hybrid_clock.cc:648] HybridClock initialized: now 1786237081696470 us; error 71 us; skew 500 ppm
I20260809 00:58:01.697067 9689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:58:01.700639 9689 webserver.cc:460] Webserver started at http://127.9.225.62:35635/ using document root <none> and password file <none>
I20260809 00:58:01.701721 9689 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:01.701889 9689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:01.702287 9689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:01.705904 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "37957345df1e4035b8421ac99bbc834e"
format_stamp: "Formatted at 2026-08-09 00:58:01 on dist-test-slave-j2vl"
server_key: "0ed4151ec14ff7fb01a8ab368ec49d10"
server_key_iv: "72254e869799b2e3fa9bef93cb5b1973"
server_key_version: "encryptionkey@0"
I20260809 00:58:01.706908 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "37957345df1e4035b8421ac99bbc834e"
format_stamp: "Formatted at 2026-08-09 00:58:01 on dist-test-slave-j2vl"
server_key: "0ed4151ec14ff7fb01a8ab368ec49d10"
server_key_iv: "72254e869799b2e3fa9bef93cb5b1973"
server_key_version: "encryptionkey@0"
I20260809 00:58:01.713721 9689 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260809 00:58:01.718032 9710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:01.719910 9689 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:58:01.720120 9689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "37957345df1e4035b8421ac99bbc834e"
format_stamp: "Formatted at 2026-08-09 00:58:01 on dist-test-slave-j2vl"
server_key: "0ed4151ec14ff7fb01a8ab368ec49d10"
server_key_iv: "72254e869799b2e3fa9bef93cb5b1973"
server_key_version: "encryptionkey@0"
I20260809 00:58:01.720360 9689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:01.736961 9689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:01.738044 9689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:01.738334 9689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:01.759720 9689 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:36169
I20260809 00:58:01.759738 9793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:36169 every 8 connection(s)
I20260809 00:58:01.761417 9689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:01.764588 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 9689
I20260809 00:58:01.764784 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:01.765259 10116 external_mini_cluster.cc:1428] Setting key 24fe3f34eb65ddd12b82811ca4eeb73a
I20260809 00:58:01.766069 9797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:01.782191 9797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e: Bootstrap starting.
I20260809 00:58:01.782670 9800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:01.767571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51428 (local address 127.9.225.62:36169)
0809 00:58:01.768831 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:01.768860 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:01.768926 (+ 66us) server_negotiation.cc:407] Connection header received
0809 00:58:01.771584 (+ 2658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:01.771625 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:01.772411 (+ 786us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:01.773020 (+ 609us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:01.773729 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:01.775902 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:01.777536 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:01.778304 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:01.778542 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:01.779941 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:01.779951 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:01.779980 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:01.780605 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:01.780691 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:01.780722 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:01.780875 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:01.781133 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 00:58:01.781524 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":405,"threads_started":1}
I20260809 00:58:01.786414 9797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:01.787587 9797 log.cc:824] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:01.791306 9797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e: No bootstrap required, opened a new log
I20260809 00:58:01.799757 9797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37957345df1e4035b8421ac99bbc834e" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36169 } }
I20260809 00:58:01.800244 9797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:01.800405 9797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37957345df1e4035b8421ac99bbc834e, State: Initialized, Role: FOLLOWER
I20260809 00:58:01.801119 9797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37957345df1e4035b8421ac99bbc834e" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36169 } }
I20260809 00:58:01.801436 9797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:01.801601 9797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:01.801810 9797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:01.804268 9797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37957345df1e4035b8421ac99bbc834e" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36169 } }
I20260809 00:58:01.804978 9797 leader_election.cc:302] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37957345df1e4035b8421ac99bbc834e; no voters:
I20260809 00:58:01.805769 9797 leader_election.cc:288] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:01.805958 9803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:01.807163 9803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [term 1 LEADER]: Becoming Leader. State: Replica: 37957345df1e4035b8421ac99bbc834e, State: Running, Role: LEADER
I20260809 00:58:01.807745 9803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37957345df1e4035b8421ac99bbc834e" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36169 } }
I20260809 00:58:01.808171 9797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:01.811208 9805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37957345df1e4035b8421ac99bbc834e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37957345df1e4035b8421ac99bbc834e" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36169 } } }
I20260809 00:58:01.811184 9806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37957345df1e4035b8421ac99bbc834e. Latest consensus state: current_term: 1 leader_uuid: "37957345df1e4035b8421ac99bbc834e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37957345df1e4035b8421ac99bbc834e" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36169 } } }
I20260809 00:58:01.811820 9805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:01.811898 9806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:01.823063 9811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:01.831775 9811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:01.843600 9811 catalog_manager.cc:1347] Generated new cluster ID: 188bf8a923f54a299f46c1b327d38961
I20260809 00:58:01.843791 9811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:01.861517 9811 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:01.863619 9811 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:01.878212 9811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e: Generated new TSK 0
I20260809 00:58:01.879558 9811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:01.889286 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.1:0
--local_ip_for_outbound_sockets=127.9.225.1
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:36169
--builtin_ntp_servers=127.9.225.20:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 00:58:02.296624 9831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:02.296990 9831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:02.297144 9831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:02.305259 9831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:02.305490 9831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:02.305681 9831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.1
I20260809 00:58:02.316583 9831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.225.1
--webserver_port=0
--tserver_master_addrs=127.9.225.62:36169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.1
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:02.318246 9831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:02.320433 9831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:02.335665 9836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:02.335672 9843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:02.336020 9837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:02.336680 9831 server_base.cc:1034] running on GCE node
I20260809 00:58:02.337419 9831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:02.338508 9831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:02.339740 9831 hybrid_clock.cc:648] HybridClock initialized: now 1786237082339673 us; error 63 us; skew 500 ppm
I20260809 00:58:02.340144 9831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:58:02.342921 9831 webserver.cc:460] Webserver started at http://127.9.225.1:41593/ using document root <none> and password file <none>
I20260809 00:58:02.343761 9831 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:02.343906 9831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:02.344307 9831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:02.347749 9831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/data/instance:
uuid: "a4c728542fa7463dadca317e9fb7bd35"
format_stamp: "Formatted at 2026-08-09 00:58:02 on dist-test-slave-j2vl"
server_key: "4840b9d39316f1a6fbdd6822eb4d4e35"
server_key_iv: "52c3b3004925a683dc48e1b068a9f2e8"
server_key_version: "encryptionkey@0"
I20260809 00:58:02.348743 9831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance:
uuid: "a4c728542fa7463dadca317e9fb7bd35"
format_stamp: "Formatted at 2026-08-09 00:58:02 on dist-test-slave-j2vl"
server_key: "4840b9d39316f1a6fbdd6822eb4d4e35"
server_key_iv: "52c3b3004925a683dc48e1b068a9f2e8"
server_key_version: "encryptionkey@0"
I20260809 00:58:02.356478 9831 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.002s
I20260809 00:58:02.362766 9856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:02.364718 9831 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:58:02.365003 9831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/wal
uuid: "a4c728542fa7463dadca317e9fb7bd35"
format_stamp: "Formatted at 2026-08-09 00:58:02 on dist-test-slave-j2vl"
server_key: "4840b9d39316f1a6fbdd6822eb4d4e35"
server_key_iv: "52c3b3004925a683dc48e1b068a9f2e8"
server_key_version: "encryptionkey@0"
I20260809 00:58:02.365329 9831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:02.403460 9831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:02.404485 9831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:02.404814 9831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:02.406061 9831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:58:02.407841 9831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:58:02.407979 9831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:02.408092 9831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:58:02.408155 9831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:02.447388 9831 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.1:36087
I20260809 00:58:02.447445 10051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.1:36087 every 8 connection(s)
I20260809 00:58:02.448873 9831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/data/info.pb
I20260809 00:58:02.452684 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 9831
I20260809 00:58:02.452893 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-0/wal/instance
I20260809 00:58:02.453559 10116 external_mini_cluster.cc:1428] Setting key 626a93f9b93cdb8cd1f74208c167641f
I20260809 00:58:02.456940 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.2:0
--local_ip_for_outbound_sockets=127.9.225.2
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:36169
--builtin_ntp_servers=127.9.225.20:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 00:58:02.469305 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:02.452860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:36169 (local address 127.9.225.1:41597)
0809 00:58:02.454159 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:02.454240 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:02.456319 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:02.457244 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:02.457255 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:02.457420 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:02.458849 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:02.458877 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:02.460045 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:02.460056 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:02.462646 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:02.462673 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:02.464333 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:02.464344 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:02.464778 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:02.465902 (+ 1124us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:58:02.465908 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:02.466002 (+ 94us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:58:02.466151 (+ 149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:02.466690 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:02.466704 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:02.466717 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 00:58:02.466983 (+ 266us) client_negotiation.cc:240] Negotiation successful
0809 00:58:02.467568 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":462,"threads_started":1}
I20260809 00:58:02.470268 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:02.452020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:41597 (local address 127.9.225.62:36169)
0809 00:58:02.452587 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:02.452592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:02.454429 (+ 1837us) server_negotiation.cc:407] Connection header received
0809 00:58:02.456648 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:02.456654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:02.456760 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:02.456897 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:02.459099 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:02.459853 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:02.462949 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:02.463776 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:02.463951 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:02.466474 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:02.466480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:02.466483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:02.466507 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:02.466555 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:02.466570 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:02.466688 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:02.469616 (+ 2928us) server_negotiation.cc:299] Negotiation successful
0809 00:58:02.469792 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":321,"threads_started":1}
I20260809 00:58:02.472126 10054 heartbeater.cc:347] Connected to a master server at 127.9.225.62:36169
I20260809 00:58:02.472586 10054 heartbeater.cc:464] Registering TS with master...
I20260809 00:58:02.473989 10054 heartbeater.cc:511] Master 127.9.225.62:36169 requested a full tablet report, sending...
I20260809 00:58:02.477288 9744 ts_manager.cc:194] Registered new tserver with Master: a4c728542fa7463dadca317e9fb7bd35 (127.9.225.1:36087)
I20260809 00:58:02.479852 9744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.225.1:41597
W20260809 00:58:02.860736 10060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:02.861120 10060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:02.861219 10060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:02.869860 10060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:02.870018 10060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:02.870174 10060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.2
I20260809 00:58:02.880312 10060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.225.2
--webserver_port=0
--tserver_master_addrs=127.9.225.62:36169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.2
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:02.882017 10060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:02.884102 10060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:02.899372 10066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:02.899348 10068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:02.899291 10070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:02.901121 10060 server_base.cc:1034] running on GCE node
I20260809 00:58:02.902336 10060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:02.903590 10060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:02.904891 10060 hybrid_clock.cc:648] HybridClock initialized: now 1786237082904819 us; error 78 us; skew 500 ppm
I20260809 00:58:02.905328 10060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:58:02.912559 10060 webserver.cc:460] Webserver started at http://127.9.225.2:43263/ using document root <none> and password file <none>
I20260809 00:58:02.913622 10060 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:02.913793 10060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:02.914260 10060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:02.917814 10060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/data/instance:
uuid: "fbbd361953294bc180ce53a00162bde8"
format_stamp: "Formatted at 2026-08-09 00:58:02 on dist-test-slave-j2vl"
server_key: "6766d9826965663ec1e3098e717ce5e8"
server_key_iv: "981b543aee0de65a252bb884bb938794"
server_key_version: "encryptionkey@0"
I20260809 00:58:02.918797 10060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance:
uuid: "fbbd361953294bc180ce53a00162bde8"
format_stamp: "Formatted at 2026-08-09 00:58:02 on dist-test-slave-j2vl"
server_key: "6766d9826965663ec1e3098e717ce5e8"
server_key_iv: "981b543aee0de65a252bb884bb938794"
server_key_version: "encryptionkey@0"
I20260809 00:58:02.925473 10060 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260809 00:58:02.929920 10080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:02.931859 10060 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260809 00:58:02.932154 10060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/wal
uuid: "fbbd361953294bc180ce53a00162bde8"
format_stamp: "Formatted at 2026-08-09 00:58:02 on dist-test-slave-j2vl"
server_key: "6766d9826965663ec1e3098e717ce5e8"
server_key_iv: "981b543aee0de65a252bb884bb938794"
server_key_version: "encryptionkey@0"
I20260809 00:58:02.932484 10060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:02.949836 10060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:02.950825 10060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:02.951150 10060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:02.952324 10060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:58:02.954298 10060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:58:02.954442 10060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:02.954604 10060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:58:02.954699 10060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:02.991468 10060 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.2:36765
I20260809 00:58:02.991537 10263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.2:36765 every 8 connection(s)
I20260809 00:58:02.993062 10060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/data/info.pb
I20260809 00:58:02.993888 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 10060
I20260809 00:58:02.994099 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/wal/instance
I20260809 00:58:02.994827 10116 external_mini_cluster.cc:1428] Setting key 4d4cf3a8434f4c14ebc923a45b56cfc2
I20260809 00:58:02.998450 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.225.3:0
--local_ip_for_outbound_sockets=127.9.225.3
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:36169
--builtin_ntp_servers=127.9.225.20:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 00:58:03.009654 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:02.996576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.2:44535 (local address 127.9.225.62:36169)
0809 00:58:02.997137 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:02.997144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:02.999134 (+ 1990us) server_negotiation.cc:407] Connection header received
0809 00:58:03.001146 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:03.001157 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:03.001269 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:03.001449 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:03.002777 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.004006 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.005979 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.006730 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.006901 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:03.008475 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:03.008484 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:03.008492 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:03.008524 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:03.008581 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:03.008644 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:03.008742 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:03.009010 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 00:58:03.009146 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":324,"threads_started":1}
I20260809 00:58:03.010825 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:02.997471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:36169 (local address 127.9.225.2:44535)
0809 00:58:02.998916 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:02.998986 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:03.000751 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:03.001694 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:03.001703 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:03.001812 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:03.002537 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:03.002559 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.004223 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.004228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:03.005769 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:03.005787 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.007002 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.007007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:03.007249 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:03.008077 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:58:03.008079 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:03.008127 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:58:03.008225 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:03.008744 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:03.008755 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:03.008763 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 00:58:03.008962 (+ 199us) client_negotiation.cc:240] Negotiation successful
0809 00:58:03.009334 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":415,"threads_started":1}
I20260809 00:58:03.012653 10265 heartbeater.cc:347] Connected to a master server at 127.9.225.62:36169
I20260809 00:58:03.013470 10265 heartbeater.cc:464] Registering TS with master...
I20260809 00:58:03.014865 10265 heartbeater.cc:511] Master 127.9.225.62:36169 requested a full tablet report, sending...
I20260809 00:58:03.016968 9744 ts_manager.cc:194] Registered new tserver with Master: fbbd361953294bc180ce53a00162bde8 (127.9.225.2:36765)
I20260809 00:58:03.018323 9744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.225.2:44535
W20260809 00:58:03.395897 10270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:03.396273 10270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:03.396376 10270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:03.404582 10270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:03.404744 10270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:03.404865 10270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.225.3
I20260809 00:58:03.415258 10270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.225.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.225.3
--webserver_port=0
--tserver_master_addrs=127.9.225.62:36169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.225.3
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:03.416738 10270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:03.418881 10270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:03.434381 10279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:03.434410 10278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:03.434968 10283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:03.436578 10270 server_base.cc:1034] running on GCE node
I20260809 00:58:03.437534 10270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:03.438707 10270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:03.439949 10270 hybrid_clock.cc:648] HybridClock initialized: now 1786237083439895 us; error 46 us; skew 500 ppm
I20260809 00:58:03.440335 10270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 00:58:03.443148 10270 webserver.cc:460] Webserver started at http://127.9.225.3:36179/ using document root <none> and password file <none>
I20260809 00:58:03.444135 10270 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:03.444286 10270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:03.444823 10270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:03.450546 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/data/instance:
uuid: "5b512fc4473b4cddb2b7d4790638da75"
format_stamp: "Formatted at 2026-08-09 00:58:03 on dist-test-slave-j2vl"
server_key: "5ceb94c72e3b2a2e50aa6f270d959f61"
server_key_iv: "da6dcb5c657563c2c784ddb7649307f9"
server_key_version: "encryptionkey@0"
I20260809 00:58:03.452105 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance:
uuid: "5b512fc4473b4cddb2b7d4790638da75"
format_stamp: "Formatted at 2026-08-09 00:58:03 on dist-test-slave-j2vl"
server_key: "5ceb94c72e3b2a2e50aa6f270d959f61"
server_key_iv: "da6dcb5c657563c2c784ddb7649307f9"
server_key_version: "encryptionkey@0"
I20260809 00:58:03.461838 10270 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.004s
I20260809 00:58:03.466445 10292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:03.468123 10270 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260809 00:58:03.468310 10270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/wal
uuid: "5b512fc4473b4cddb2b7d4790638da75"
format_stamp: "Formatted at 2026-08-09 00:58:03 on dist-test-slave-j2vl"
server_key: "5ceb94c72e3b2a2e50aa6f270d959f61"
server_key_iv: "da6dcb5c657563c2c784ddb7649307f9"
server_key_version: "encryptionkey@0"
I20260809 00:58:03.468544 10270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:03.483889 10270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:03.484859 10270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:03.485168 10270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:03.485554 10054 heartbeater.cc:503] Master 127.9.225.62:36169 was elected leader, sending a full tablet report...
I20260809 00:58:03.486580 10270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 00:58:03.488324 10270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 00:58:03.488466 10270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:03.488620 10270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 00:58:03.488713 10270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:03.529050 10270 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.3:40945
I20260809 00:58:03.529088 10497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.3:40945 every 8 connection(s)
I20260809 00:58:03.530759 10270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/data/info.pb
I20260809 00:58:03.533094 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 10270
I20260809 00:58:03.533301 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/wal/instance
I20260809 00:58:03.533831 10116 external_mini_cluster.cc:1428] Setting key 76c1beed041100047a80450d27bfb54b
I20260809 00:58:03.547045 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:03.534700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.3:33933 (local address 127.9.225.62:36169)
0809 00:58:03.535451 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:03.535456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:03.537084 (+ 1628us) server_negotiation.cc:407] Connection header received
0809 00:58:03.539217 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:03.539223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:03.539322 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:03.539466 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:03.540786 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.541701 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.543547 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.544003 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.544162 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:03.545981 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:03.545987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:03.545991 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:03.546016 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:03.546064 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:03.546078 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:03.546186 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:03.546464 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 00:58:03.546594 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":299,"threads_started":1}
I20260809 00:58:03.548476 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:03.535573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:36169 (local address 127.9.225.3:33933)
0809 00:58:03.536888 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:03.536961 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:03.538887 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:03.539768 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:03.539778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:03.539886 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:03.540579 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:03.540599 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.541884 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.541890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:03.543344 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:03.543360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.544291 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.544300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:03.544592 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:03.545524 (+ 932us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 00:58:03.545527 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:03.545581 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 00:58:03.545689 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:03.546191 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:03.546206 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:03.546216 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 00:58:03.546458 (+ 242us) client_negotiation.cc:240] Negotiation successful
0809 00:58:03.547005 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":379,"threads_started":1}
I20260809 00:58:03.550228 10498 heartbeater.cc:347] Connected to a master server at 127.9.225.62:36169
I20260809 00:58:03.550642 10498 heartbeater.cc:464] Registering TS with master...
I20260809 00:58:03.551576 10498 heartbeater.cc:511] Master 127.9.225.62:36169 requested a full tablet report, sending...
I20260809 00:58:03.553310 9744 ts_manager.cc:194] Registered new tserver with Master: 5b512fc4473b4cddb2b7d4790638da75 (127.9.225.3:40945)
I20260809 00:58:03.554289 9744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.225.3:33933
I20260809 00:58:03.563599 10116 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 00:58:03.575579 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:03.568550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.9.225.62:36169)
0809 00:58:03.568809 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:03.568814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:03.569217 (+ 403us) server_negotiation.cc:407] Connection header received
0809 00:58:03.569446 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:03.569454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:03.569557 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:03.569660 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:03.570294 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.571157 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.572816 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.573493 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.573614 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:03.574827 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:03.574832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:03.574836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:03.574864 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:03.574904 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:03.574917 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:03.575004 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:03.575176 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 00:58:03.575289 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 00:58:03.581147 9744 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 00:58:03.588658 9744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 00:58:03.616207 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:03.603172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:40945 (local address 127.0.0.1:40978)
0809 00:58:03.604076 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:03.604123 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:03.604343 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:03.605797 (+ 1454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:03.605807 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:03.605856 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:58:03.606450 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:03.606471 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.608944 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.608954 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:03.611998 (+ 3044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:03.612026 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.615104 (+ 3078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.615114 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:03.615378 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:03.615460 (+ 82us) client_negotiation.cc:769] Sending connection context
0809 00:58:03.615611 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 00:58:03.615837 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":361,"threads_started":1}
I20260809 00:58:03.616938 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:03.603338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40978 (local address 127.9.225.3:40945)
0809 00:58:03.604294 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:03.604303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:03.604339 (+ 36us) server_negotiation.cc:407] Connection header received
0809 00:58:03.604811 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:03.604823 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:03.605100 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:03.605431 (+ 331us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:58:03.606727 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.608707 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.612377 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.614161 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.614347 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:03.614697 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:03.616185 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0809 00:58:03.616544 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":591,"threads_started":1}
I20260809 00:58:03.622575 10371 tablet_service.cc:1467] Processing CreateTablet for tablet e23e48aed54441029c5429ac7f0be8e3 (DEFAULT_TABLE table=test-table [id=2675cf7162cf4ca09e8d27d1ce4aef3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:58:03.624179 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:03.606083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.9.225.2:36765)
0809 00:58:03.613559 (+ 7476us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:03.613570 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:03.613607 (+ 37us) server_negotiation.cc:407] Connection header received
0809 00:58:03.613738 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:03.613763 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:03.614078 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:03.614366 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:58:03.615184 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.617648 (+ 2464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.621139 (+ 3491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.622813 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.623025 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:03.623305 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:03.623635 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 00:58:03.623820 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7293,"thread_start_us":7141,"threads_started":1}
I20260809 00:58:03.624185 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:03.605157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:36765 (local address 127.0.0.1:34172)
0809 00:58:03.612923 (+ 7766us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:03.612941 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:03.613045 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:03.614361 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:03.614366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:03.614400 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:58:03.614848 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:03.614860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.617899 (+ 3039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.617908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:03.620726 (+ 2818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:03.620761 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.623095 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.623102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:03.623295 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:03.623342 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 00:58:03.623556 (+ 214us) client_negotiation.cc:240] Negotiation successful
0809 00:58:03.623744 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7631,"mutex_wait_us":158,"thread_start_us":7103,"threads_started":1}
I20260809 00:58:03.624742 10371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e23e48aed54441029c5429ac7f0be8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:03.624908 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:03.604566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:36087 (local address 127.0.0.1:35326)
0809 00:58:03.612589 (+ 8023us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:03.612619 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:03.612800 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:03.614851 (+ 2051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:03.614855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:03.614884 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:58:03.615583 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:03.615603 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.618196 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.618210 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:03.621334 (+ 3124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:03.621402 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.623942 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.623949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:03.624184 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:03.624228 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 00:58:03.624330 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 00:58:03.624489 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7857,"mutex_wait_us":72,"thread_start_us":8330,"threads_started":1}
I20260809 00:58:03.625710 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:03.604938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.9.225.1:36087)
0809 00:58:03.613655 (+ 8717us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:03.613663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:03.613688 (+ 25us) server_negotiation.cc:407] Connection header received
0809 00:58:03.613787 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:03.613793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:03.614078 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:03.614366 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:58:03.615840 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.617927 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.621771 (+ 3844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:03.623319 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:03.623487 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:03.623768 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:03.624936 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0809 00:58:03.625254 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8524,"thread_start_us":8569,"threads_started":1}
I20260809 00:58:03.628960 10171 tablet_service.cc:1467] Processing CreateTablet for tablet e23e48aed54441029c5429ac7f0be8e3 (DEFAULT_TABLE table=test-table [id=2675cf7162cf4ca09e8d27d1ce4aef3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:58:03.629896 9942 tablet_service.cc:1467] Processing CreateTablet for tablet e23e48aed54441029c5429ac7f0be8e3 (DEFAULT_TABLE table=test-table [id=2675cf7162cf4ca09e8d27d1ce4aef3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 00:58:03.630825 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e23e48aed54441029c5429ac7f0be8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:03.631734 9942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e23e48aed54441029c5429ac7f0be8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:03.644558 10526 tablet_bootstrap.cc:492] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75: Bootstrap starting.
I20260809 00:58:03.648315 10527 tablet_bootstrap.cc:492] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8: Bootstrap starting.
I20260809 00:58:03.650275 10529 tablet_bootstrap.cc:492] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35: Bootstrap starting.
I20260809 00:58:03.651643 10526 tablet_bootstrap.cc:654] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:03.652815 10526 log.cc:824] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:03.653100 10527 tablet_bootstrap.cc:654] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:03.654611 10527 log.cc:824] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:03.654997 10529 tablet_bootstrap.cc:654] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:03.656474 10529 log.cc:824] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:03.658878 10526 tablet_bootstrap.cc:492] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75: No bootstrap required, opened a new log
I20260809 00:58:03.659106 10527 tablet_bootstrap.cc:492] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8: No bootstrap required, opened a new log
I20260809 00:58:03.659566 10526 ts_tablet_manager.cc:1397] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260809 00:58:03.659652 10527 ts_tablet_manager.cc:1397] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260809 00:58:03.664342 10529 tablet_bootstrap.cc:492] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35: No bootstrap required, opened a new log
I20260809 00:58:03.664718 10529 ts_tablet_manager.cc:1397] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260809 00:58:03.672461 10526 raft_consensus.cc:348] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b512fc4473b4cddb2b7d4790638da75" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 40945 } } peers { permanent_uuid: "a4c728542fa7463dadca317e9fb7bd35" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 36087 } } peers { permanent_uuid: "fbbd361953294bc180ce53a00162bde8" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36765 } }
I20260809 00:58:03.672252 10527 raft_consensus.cc:348] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b512fc4473b4cddb2b7d4790638da75" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 40945 } } peers { permanent_uuid: "a4c728542fa7463dadca317e9fb7bd35" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 36087 } } peers { permanent_uuid: "fbbd361953294bc180ce53a00162bde8" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36765 } }
I20260809 00:58:03.673108 10527 raft_consensus.cc:374] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:03.673108 10526 raft_consensus.cc:374] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:03.673246 10526 raft_consensus.cc:729] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b512fc4473b4cddb2b7d4790638da75, State: Initialized, Role: FOLLOWER
I20260809 00:58:03.673246 10527 raft_consensus.cc:729] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbbd361953294bc180ce53a00162bde8, State: Initialized, Role: FOLLOWER
I20260809 00:58:03.674257 10526 consensus_queue.cc:260] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b512fc4473b4cddb2b7d4790638da75" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 40945 } } peers { permanent_uuid: "a4c728542fa7463dadca317e9fb7bd35" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 36087 } } peers { permanent_uuid: "fbbd361953294bc180ce53a00162bde8" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36765 } }
I20260809 00:58:03.674258 10527 consensus_queue.cc:260] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b512fc4473b4cddb2b7d4790638da75" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 40945 } } peers { permanent_uuid: "a4c728542fa7463dadca317e9fb7bd35" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 36087 } } peers { permanent_uuid: "fbbd361953294bc180ce53a00162bde8" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36765 } }
I20260809 00:58:03.674293 10529 raft_consensus.cc:348] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b512fc4473b4cddb2b7d4790638da75" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 40945 } } peers { permanent_uuid: "a4c728542fa7463dadca317e9fb7bd35" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 36087 } } peers { permanent_uuid: "fbbd361953294bc180ce53a00162bde8" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36765 } }
I20260809 00:58:03.674705 10529 raft_consensus.cc:374] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:03.674847 10529 raft_consensus.cc:729] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c728542fa7463dadca317e9fb7bd35, State: Initialized, Role: FOLLOWER
I20260809 00:58:03.675698 10265 heartbeater.cc:503] Master 127.9.225.62:36169 was elected leader, sending a full tablet report...
I20260809 00:58:03.675696 10529 consensus_queue.cc:260] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b512fc4473b4cddb2b7d4790638da75" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 40945 } } peers { permanent_uuid: "a4c728542fa7463dadca317e9fb7bd35" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 36087 } } peers { permanent_uuid: "fbbd361953294bc180ce53a00162bde8" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36765 } }
I20260809 00:58:03.676870 10527 ts_tablet_manager.cc:1428] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260809 00:58:03.679495 10498 heartbeater.cc:503] Master 127.9.225.62:36169 was elected leader, sending a full tablet report...
I20260809 00:58:03.679981 10529 ts_tablet_manager.cc:1428] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260809 00:58:03.681568 10526 ts_tablet_manager.cc:1428] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75: Time spent starting tablet: real 0.022s user 0.013s sys 0.009s
W20260809 00:58:03.701891 10056 tablet.cc:2369] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:58:03.746280 10266 tablet.cc:2369] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 00:58:03.783216 10499 tablet.cc:2369] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:58:04.020815 10540 raft_consensus.cc:482] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 00:58:04.021236 10540 raft_consensus.cc:504] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b512fc4473b4cddb2b7d4790638da75" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 40945 } } peers { permanent_uuid: "a4c728542fa7463dadca317e9fb7bd35" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 36087 } } peers { permanent_uuid: "fbbd361953294bc180ce53a00162bde8" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36765 } }
I20260809 00:58:04.023244 10540 leader_election.cc:288] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b512fc4473b4cddb2b7d4790638da75 (127.9.225.3:40945), fbbd361953294bc180ce53a00162bde8 (127.9.225.2:36765)
I20260809 00:58:04.030483 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.023867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:49461 (local address 127.9.225.2:36765)
0809 00:58:04.024082 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:04.024090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:04.024919 (+ 829us) server_negotiation.cc:407] Connection header received
0809 00:58:04.025130 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:04.025135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:04.025262 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:04.025386 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:58:04.025995 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.026761 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.028586 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.029506 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.029650 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:04.029754 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:04.030057 (+ 303us) server_negotiation.cc:299] Negotiation successful
0809 00:58:04.030192 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 00:58:04.030560 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.023690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.2:36765 (local address 127.9.225.1:49461)
0809 00:58:04.024802 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:04.024818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:04.024983 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:04.025410 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:04.025414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:04.025444 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:58:04.025822 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:04.025830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.026910 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.026917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:04.028384 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:04.028396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.029705 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.029713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:04.029887 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:04.029912 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 00:58:04.030026 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 00:58:04.030175 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"thread_start_us":762,"threads_started":1}
I20260809 00:58:04.030546 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.023630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.225.1:59043 (local address 127.9.225.3:40945)
0809 00:58:04.024083 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:04.024090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:04.024596 (+ 506us) server_negotiation.cc:407] Connection header received
0809 00:58:04.024847 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:04.024857 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:04.024929 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:04.025034 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 00:58:04.025718 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.026559 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.028514 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.029587 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.029740 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:04.029854 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:04.030140 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 00:58:04.030275 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322}
I20260809 00:58:04.030885 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.023533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.3:40945 (local address 127.9.225.1:59043)
0809 00:58:04.024465 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:04.024484 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:04.024694 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:04.025055 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:04.025059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:04.025084 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 00:58:04.025561 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:04.025574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.026697 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.026703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:04.028267 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:04.028285 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.029796 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.029803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:04.029968 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:04.029992 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 00:58:04.030106 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 00:58:04.030257 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":5,"thread_start_us":444,"threads_started":1}
I20260809 00:58:04.031860 10200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e23e48aed54441029c5429ac7f0be8e3" candidate_uuid: "a4c728542fa7463dadca317e9fb7bd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbd361953294bc180ce53a00162bde8" is_pre_election: true
I20260809 00:58:04.031983 10406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e23e48aed54441029c5429ac7f0be8e3" candidate_uuid: "a4c728542fa7463dadca317e9fb7bd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b512fc4473b4cddb2b7d4790638da75" is_pre_election: true
I20260809 00:58:04.032488 10200 raft_consensus.cc:2454] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4c728542fa7463dadca317e9fb7bd35 in term 0.
I20260809 00:58:04.032524 10406 raft_consensus.cc:2454] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4c728542fa7463dadca317e9fb7bd35 in term 0.
I20260809 00:58:04.033236 9859 leader_election.cc:302] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b512fc4473b4cddb2b7d4790638da75, a4c728542fa7463dadca317e9fb7bd35; no voters:
I20260809 00:58:04.033761 10540 raft_consensus.cc:2781] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 00:58:04.033941 10540 raft_consensus.cc:482] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 00:58:04.034070 10540 raft_consensus.cc:3037] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:04.036374 10540 raft_consensus.cc:504] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b512fc4473b4cddb2b7d4790638da75" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 40945 } } peers { permanent_uuid: "a4c728542fa7463dadca317e9fb7bd35" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 36087 } } peers { permanent_uuid: "fbbd361953294bc180ce53a00162bde8" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36765 } }
I20260809 00:58:04.037185 10540 leader_election.cc:288] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [CANDIDATE]: Term 1 election: Requested vote from peers 5b512fc4473b4cddb2b7d4790638da75 (127.9.225.3:40945), fbbd361953294bc180ce53a00162bde8 (127.9.225.2:36765)
I20260809 00:58:04.037829 10200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e23e48aed54441029c5429ac7f0be8e3" candidate_uuid: "a4c728542fa7463dadca317e9fb7bd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbbd361953294bc180ce53a00162bde8"
I20260809 00:58:04.037832 10406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e23e48aed54441029c5429ac7f0be8e3" candidate_uuid: "a4c728542fa7463dadca317e9fb7bd35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b512fc4473b4cddb2b7d4790638da75"
I20260809 00:58:04.038089 10406 raft_consensus.cc:3037] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:04.038089 10200 raft_consensus.cc:3037] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:04.040513 10406 raft_consensus.cc:2454] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4c728542fa7463dadca317e9fb7bd35 in term 1.
I20260809 00:58:04.040514 10200 raft_consensus.cc:2454] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4c728542fa7463dadca317e9fb7bd35 in term 1.
I20260809 00:58:04.041117 9860 leader_election.cc:302] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4c728542fa7463dadca317e9fb7bd35, fbbd361953294bc180ce53a00162bde8; no voters:
I20260809 00:58:04.041532 10540 raft_consensus.cc:2781] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:04.042089 10540 raft_consensus.cc:686] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 1 LEADER]: Becoming Leader. State: Replica: a4c728542fa7463dadca317e9fb7bd35, State: Running, Role: LEADER
I20260809 00:58:04.042650 10540 consensus_queue.cc:237] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b512fc4473b4cddb2b7d4790638da75" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 40945 } } peers { permanent_uuid: "a4c728542fa7463dadca317e9fb7bd35" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 36087 } } peers { permanent_uuid: "fbbd361953294bc180ce53a00162bde8" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36765 } }
I20260809 00:58:04.047720 9744 catalog_manager.cc:5495] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4c728542fa7463dadca317e9fb7bd35 (127.9.225.1). New cstate: current_term: 1 leader_uuid: "a4c728542fa7463dadca317e9fb7bd35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b512fc4473b4cddb2b7d4790638da75" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4c728542fa7463dadca317e9fb7bd35" member_type: VOTER last_known_addr { host: "127.9.225.1" port: 36087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbbd361953294bc180ce53a00162bde8" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36765 } health_report { overall_health: UNKNOWN } } }
I20260809 00:58:04.079835 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.074085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.9.225.1:36087)
0809 00:58:04.074332 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:04.074339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:04.074385 (+ 46us) server_negotiation.cc:407] Connection header received
0809 00:58:04.074491 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:04.074495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:04.074592 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:04.074797 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:58:04.075224 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.075954 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.077470 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.077916 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.078061 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:04.078626 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:58:04.079211 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:58:04.079404 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:04.079458 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 00:58:04.079549 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 00:58:04.089267 10200 raft_consensus.cc:1261] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8 [term 1 FOLLOWER]: Refusing update from remote peer a4c728542fa7463dadca317e9fb7bd35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:58:04.089310 10406 raft_consensus.cc:1261] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75 [term 1 FOLLOWER]: Refusing update from remote peer a4c728542fa7463dadca317e9fb7bd35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:58:04.090561 10544 consensus_queue.cc:1035] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b512fc4473b4cddb2b7d4790638da75" member_type: VOTER last_known_addr { host: "127.9.225.3" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:58:04.090982 10540 consensus_queue.cc:1035] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbbd361953294bc180ce53a00162bde8" member_type: VOTER last_known_addr { host: "127.9.225.2" port: 36765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:58:04.105650 10554 mvcc.cc:205] Tried to move back new op lower bound from 7316427096418381824 to 7316427096243011584. Current Snapshot: MvccSnapshot[applied={T|T < 7316427096418381824}]
I20260809 00:58:04.107503 10555 mvcc.cc:205] Tried to move back new op lower bound from 7316427096418381824 to 7316427096243011584. Current Snapshot: MvccSnapshot[applied={T|T < 7316427096418381824}]
I20260809 00:58:04.123495 9744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:58:04.123826 9744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 00:58:04.128217 9744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37957345df1e4035b8421ac99bbc834e: Sending DeleteTablet for 3 replicas of tablet e23e48aed54441029c5429ac7f0be8e3
I20260809 00:58:04.129835 10371 tablet_service.cc:1514] Processing DeleteTablet for tablet e23e48aed54441029c5429ac7f0be8e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:58:04 UTC) from {username='slave'} at 127.0.0.1:40978
I20260809 00:58:04.129832 10171 tablet_service.cc:1514] Processing DeleteTablet for tablet e23e48aed54441029c5429ac7f0be8e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:58:04 UTC) from {username='slave'} at 127.0.0.1:34172
I20260809 00:58:04.130707 9942 tablet_service.cc:1514] Processing DeleteTablet for tablet e23e48aed54441029c5429ac7f0be8e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 00:58:04 UTC) from {username='slave'} at 127.0.0.1:35326
I20260809 00:58:04.130964 10569 tablet_replica.cc:330] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75: stopping tablet replica
I20260809 00:58:04.131542 10569 raft_consensus.cc:2229] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:58:04.131770 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 9831
I20260809 00:58:04.132315 10569 raft_consensus.cc:2258] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:58:04.135119 10569 ts_tablet_manager.cc:1905] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:58:04.144492 10569 ts_tablet_manager.cc:1918] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:58:04.144912 10569 log.cc:1196] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-2/wal/wals/e23e48aed54441029c5429ac7f0be8e3
I20260809 00:58:04.145821 10569 ts_tablet_manager.cc:1939] T e23e48aed54441029c5429ac7f0be8e3 P 5b512fc4473b4cddb2b7d4790638da75: Deleting consensus metadata
I20260809 00:58:04.148962 9712 catalog_manager.cc:4841] TS 5b512fc4473b4cddb2b7d4790638da75 (127.9.225.3:40945): tablet e23e48aed54441029c5429ac7f0be8e3 (table test-table [id=2675cf7162cf4ca09e8d27d1ce4aef3d]) successfully deleted
I20260809 00:58:04.150292 10572 tablet_replica.cc:330] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35: stopping tablet replica
I20260809 00:58:04.150887 10572 raft_consensus.cc:2229] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 1 LEADER]: Raft consensus shutting down.
I20260809 00:58:04.151654 10572 raft_consensus.cc:2258] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:58:04.153956 10571 tablet_replica.cc:330] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8: stopping tablet replica
I20260809 00:58:04.154531 10571 raft_consensus.cc:2229] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 00:58:04.154881 10572 ts_tablet_manager.cc:1905] T e23e48aed54441029c5429ac7f0be8e3 P a4c728542fa7463dadca317e9fb7bd35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:58:04.155301 10571 raft_consensus.cc:2258] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 00:58:04.158053 10571 ts_tablet_manager.cc:1905] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 00:58:04.159943 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.136645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.9.225.1:36087)
0809 00:58:04.136873 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:04.136884 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:04.150667 (+ 13783us) server_negotiation.cc:407] Connection header received
0809 00:58:04.150922 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:04.150928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:04.151076 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:04.151208 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:04.151723 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.153012 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.155142 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.155864 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.155978 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:04.157957 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:04.157973 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:04.158003 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:04.158884 (+ 881us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:04.158962 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:04.158994 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:04.159100 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:04.159378 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 00:58:04.159536 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 00:58:04.161374 10039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:04.166723 10571 ts_tablet_manager.cc:1918] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 00:58:04.166960 10571 log.cc:1196] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8: Deleting WAL directory at /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786236719337175-10116-0/minicluster-data/ts-1/wal/wals/e23e48aed54441029c5429ac7f0be8e3
I20260809 00:58:04.167536 10571 ts_tablet_manager.cc:1939] T e23e48aed54441029c5429ac7f0be8e3 P fbbd361953294bc180ce53a00162bde8: Deleting consensus metadata
I20260809 00:58:04.169821 9713 catalog_manager.cc:4841] TS fbbd361953294bc180ce53a00162bde8 (127.9.225.2:36765): tablet e23e48aed54441029c5429ac7f0be8e3 (table test-table [id=2675cf7162cf4ca09e8d27d1ce4aef3d]) successfully deleted
I20260809 00:58:04.264776 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 9831
W20260809 00:58:04.293205 9712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.225.1:36087 (error 108)
I20260809 00:58:04.294076 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 10060
I20260809 00:58:04.295692 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.294770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:36087 (local address 127.0.0.1:35366)
0809 00:58:04.295374 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:04.295501 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":355,"threads_started":1}
W20260809 00:58:04.296056 9712 catalog_manager.cc:4584] TS a4c728542fa7463dadca317e9fb7bd35 (127.9.225.1:36087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e23e48aed54441029c5429ac7f0be8e3: Network error: Client connection negotiation failed: client connection to 127.9.225.1:36087: connect: Connection refused (error 111)
I20260809 00:58:04.301616 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.294935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.9.225.2:36765)
0809 00:58:04.295134 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:04.295140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:04.295163 (+ 23us) server_negotiation.cc:407] Connection header received
0809 00:58:04.295389 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:04.295394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:04.295480 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:04.295596 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:04.296138 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.296961 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.298350 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.298777 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.298881 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:04.300031 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:04.300041 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:04.300058 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:04.300718 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:04.300769 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:04.300789 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:04.300884 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:04.301067 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 00:58:04.301169 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 00:58:04.302292 10257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:04.350207 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.349682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:36087 (local address 127.0.0.1:35374)
0809 00:58:04.349930 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:04.350040 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260809 00:58:04.351759 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.351264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:36087 (local address 127.0.0.1:35386)
0809 00:58:04.351494 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:04.351577 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 00:58:04.396884 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 10060
I20260809 00:58:04.412322 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.411827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:36087 (local address 127.0.0.1:35388)
0809 00:58:04.412063 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:04.412169 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 00:58:04.413496 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.413038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:36087 (local address 127.0.0.1:35394)
0809 00:58:04.413197 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:04.413279 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 00:58:04.429824 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 10270
I20260809 00:58:04.437635 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.430788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.9.225.3:40945)
0809 00:58:04.431008 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:04.431013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:04.431034 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:58:04.431209 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:04.431215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:04.431333 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:04.431433 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:04.431951 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.432678 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.434187 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:04.434632 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:04.434759 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:04.435998 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:04.436008 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:04.436026 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:04.436652 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:04.436707 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:04.436728 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:04.436832 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:04.437054 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 00:58:04.437207 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260809 00:58:04.438405 10484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:04.494354 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.493833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:36087 (local address 127.0.0.1:35408)
0809 00:58:04.494084 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:04.494188 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 00:58:04.495716 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.495216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:36087 (local address 127.0.0.1:35422)
0809 00:58:04.495458 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:04.495535 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260809 00:58:04.534557 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 10270
I20260809 00:58:04.559197 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 9689
I20260809 00:58:04.560520 9785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:04.678999 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 9689
I20260809 00:58:04.679044 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:04.678476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.1:36087 (local address 127.0.0.1:35424)
0809 00:58:04.678693 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:04.678834 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.1:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
2026-08-09T00:58:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3543 ms)
[----------] 6 tests from EncryptionPolicyTest (27242 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:04 dist-test-slave-j2vl krb5kdc[10598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:04 dist-test-slave-j2vl krb5kdc[10598](info): set up 2 sockets
Aug 09 00:58:04 dist-test-slave-j2vl krb5kdc[10598](info): commencing operation
krb5kdc: starting...
W20260809 00:58:06.150981 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.407s user 0.002s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:06 dist-test-slave-j2vl krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 00:58:06.531445 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:06.914606 10618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:06.915081 10618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:06.915268 10618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:06.923012 10618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:06.923233 10618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:06.923352 10618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:06.923468 10618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:06.923560 10618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:06.934865 10618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:06.936543 10618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:06.938151 10618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:06.954865 10628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:06.954906 10631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:06.954959 10627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:06.955935 10618 server_base.cc:1034] running on GCE node
I20260809 00:58:06.957201 10618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:06.959615 10618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:06.960927 10618 hybrid_clock.cc:648] HybridClock initialized: now 1786237086960879 us; error 60 us; skew 500 ppm
I20260809 00:58:06.961592 10618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:06 dist-test-slave-j2vl krb5kdc[10598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:06.966337 10618 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 00:58:06.968837 10618 webserver.cc:460] Webserver started at http://127.0.0.1:37493/ using document root <none> and password file <none>
I20260809 00:58:06.969897 10618 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:06.970019 10618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:06.970328 10618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:06.973699 10618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "0a69464591ee43068f1adff89e6697c0"
format_stamp: "Formatted at 2026-08-09 00:58:06 on dist-test-slave-j2vl"
server_key: "a44ee33c5a55505fe79a7a8da48421c0"
server_key_iv: "f16a34a44394e3fecf9311c99c2b868f"
server_key_version: "encryptionkey@0"
I20260809 00:58:06.974498 10618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "0a69464591ee43068f1adff89e6697c0"
format_stamp: "Formatted at 2026-08-09 00:58:06 on dist-test-slave-j2vl"
server_key: "a44ee33c5a55505fe79a7a8da48421c0"
server_key_iv: "f16a34a44394e3fecf9311c99c2b868f"
server_key_version: "encryptionkey@0"
I20260809 00:58:06.980610 10618 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260809 00:58:06.984879 10647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:06.986914 10618 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260809 00:58:06.987164 10618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "0a69464591ee43068f1adff89e6697c0"
format_stamp: "Formatted at 2026-08-09 00:58:06 on dist-test-slave-j2vl"
server_key: "a44ee33c5a55505fe79a7a8da48421c0"
server_key_iv: "f16a34a44394e3fecf9311c99c2b868f"
server_key_version: "encryptionkey@0"
I20260809 00:58:06.987380 10618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:07.021436 10618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:07.026942 10618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:07.027241 10618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:07.047816 10618 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42613
I20260809 00:58:07.047842 10722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42613 every 8 connection(s)
I20260809 00:58:07.049512 10618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:07.053941 10723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:07.059710 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 10618
I20260809 00:58:07.059965 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:07.060866 10116 external_mini_cluster.cc:1428] Setting key 8e64c916707f7a75cdb050a78eae0bea
I20260809 00:58:07.067673 10723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0: Bootstrap starting.
I20260809 00:58:07.072202 10723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:07.073695 10723 log.cc:824] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:07.078068 10723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0: No bootstrap required, opened a new log
Aug 09 00:58:07 dist-test-slave-j2vl krb5kdc[10598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 00:58:07.087607 10723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a69464591ee43068f1adff89e6697c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } }
I20260809 00:58:07.088071 10723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:07.088246 10723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69464591ee43068f1adff89e6697c0, State: Initialized, Role: FOLLOWER
I20260809 00:58:07.089048 10723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a69464591ee43068f1adff89e6697c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } }
I20260809 00:58:07.089406 10723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:07.089603 10723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:07.089825 10723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:07.090425 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:07.063262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49389 (local address 127.0.0.1:42613)
0809 00:58:07.064316 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:07.064330 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:07.064386 (+ 56us) server_negotiation.cc:407] Connection header received
0809 00:58:07.066442 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:07.066476 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:07.067016 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:07.067420 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:07.069681 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:07.071152 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:07.073138 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:07.074078 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:07.074235 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:07.079847 (+ 5612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:07.079858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:07.079876 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:07.079927 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:07.084795 (+ 4868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:07.085993 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:07.086006 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:07.086017 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:07.086228 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:07.086818 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:07.086822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:07.086825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:07.087492 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:07.087894 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:07.088505 (+ 611us) server_negotiation.cc:299] Negotiation successful
0809 00:58:07.088939 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":501,"threads_started":1}
I20260809 00:58:07.093084 10723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a69464591ee43068f1adff89e6697c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } }
I20260809 00:58:07.093761 10723 leader_election.cc:302] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a69464591ee43068f1adff89e6697c0; no voters:
I20260809 00:58:07.094481 10723 leader_election.cc:288] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:07.094655 10732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:07.095400 10732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [term 1 LEADER]: Becoming Leader. State: Replica: 0a69464591ee43068f1adff89e6697c0, State: Running, Role: LEADER
I20260809 00:58:07.096174 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:07.096196 10732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a69464591ee43068f1adff89e6697c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } }
I20260809 00:58:07.097886 10723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:07.099633 10733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a69464591ee43068f1adff89e6697c0. Latest consensus state: current_term: 1 leader_uuid: "0a69464591ee43068f1adff89e6697c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a69464591ee43068f1adff89e6697c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } } }
I20260809 00:58:07.099596 10734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a69464591ee43068f1adff89e6697c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a69464591ee43068f1adff89e6697c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } } }
I20260809 00:58:07.100199 10734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:07.100200 10733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:07.101418 10739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:07.122285 10739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:07.135579 10739 catalog_manager.cc:1347] Generated new cluster ID: 29d98da0a9f54208aa68b971532d2642
I20260809 00:58:07.135774 10739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:07.141721 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:07.106123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36421 (local address 127.0.0.1:42613)
0809 00:58:07.111117 (+ 4994us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:07.111128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:07.122272 (+ 11144us) server_negotiation.cc:407] Connection header received
0809 00:58:07.122611 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:07.122618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:07.122716 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:07.122850 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:07.125418 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:07.126674 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:07.128613 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:07.129317 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:07.129477 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:07.133475 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:07.133482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:07.133487 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:07.133540 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:07.137855 (+ 4315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:07.139062 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:07.139069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:07.139073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:07.139229 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:07.139768 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:07.139773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:07.139777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:07.140195 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:07.140432 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:07.141111 (+ 679us) server_negotiation.cc:299] Negotiation successful
0809 00:58:07.141314 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4769}
I20260809 00:58:07.165668 10739 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:07.167752 10739 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:07.184237 10739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a69464591ee43068f1adff89e6697c0: Generated new TSK 0
I20260809 00:58:07.185732 10739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:07.192898 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 10618
I20260809 00:58:07.194561 10717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:07.291448 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 10618
W20260809 00:58:07.320190 10594 connection.cc:535] client connection to 127.0.0.1:42613 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42613): Connection reset by peer (error 104)
2026-08-09T00:58:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2597 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:07 dist-test-slave-j2vl krb5kdc[10775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:07 dist-test-slave-j2vl krb5kdc[10775](info): set up 2 sockets
Aug 09 00:58:07 dist-test-slave-j2vl krb5kdc[10775](info): commencing operation
krb5kdc: starting...
W20260809 00:58:08.701283 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.361s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:08 dist-test-slave-j2vl krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 00:58:09.081404 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:09.481992 10798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:09.482419 10798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:09.482596 10798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:09.490478 10798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:09.490684 10798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:09.490792 10798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:09.490926 10798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:09.491010 10798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:09.502619 10798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:09.504276 10798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:09.505930 10798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:09.519852 10810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:09.520004 10806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:09.520035 10812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:09.520725 10798 server_base.cc:1034] running on GCE node
I20260809 00:58:09.521996 10798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:09.523844 10798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:09.525091 10798 hybrid_clock.cc:648] HybridClock initialized: now 1786237089525044 us; error 42 us; skew 500 ppm
I20260809 00:58:09.525588 10798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:09 dist-test-slave-j2vl krb5kdc[10775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:09.534343 10798 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 00:58:09.536391 10798 webserver.cc:460] Webserver started at http://127.0.0.1:46195/ using document root <none> and password file <none>
I20260809 00:58:09.537236 10798 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:09.537446 10798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:09.537840 10798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:09.541316 10798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "17fb30ea1a4a4a85a5925bc51c5315f1"
format_stamp: "Formatted at 2026-08-09 00:58:09 on dist-test-slave-j2vl"
server_key: "6abb81f6a24c3ce145cbab72cc5497bc"
server_key_iv: "c5d53223c79aa627d4b69ed7b9ee4f92"
server_key_version: "encryptionkey@0"
I20260809 00:58:09.542304 10798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "17fb30ea1a4a4a85a5925bc51c5315f1"
format_stamp: "Formatted at 2026-08-09 00:58:09 on dist-test-slave-j2vl"
server_key: "6abb81f6a24c3ce145cbab72cc5497bc"
server_key_iv: "c5d53223c79aa627d4b69ed7b9ee4f92"
server_key_version: "encryptionkey@0"
I20260809 00:58:09.548349 10798 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 00:58:09.552646 10823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:09.554386 10798 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260809 00:58:09.554628 10798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "17fb30ea1a4a4a85a5925bc51c5315f1"
format_stamp: "Formatted at 2026-08-09 00:58:09 on dist-test-slave-j2vl"
server_key: "6abb81f6a24c3ce145cbab72cc5497bc"
server_key_iv: "c5d53223c79aa627d4b69ed7b9ee4f92"
server_key_version: "encryptionkey@0"
I20260809 00:58:09.554898 10798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:09.572865 10798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:09.577975 10798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:09.578379 10798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:09.599749 10798 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42723
I20260809 00:58:09.599795 10906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42723 every 8 connection(s)
I20260809 00:58:09.601326 10798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:09.605696 10910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:09.609974 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 10798
I20260809 00:58:09.610209 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:09.610849 10116 external_mini_cluster.cc:1428] Setting key 4091abdc886616cb6fe18158e67ebd96
I20260809 00:58:09.619817 10910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1: Bootstrap starting.
I20260809 00:58:09.623818 10910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:09.625211 10910 log.cc:824] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:09.625964 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:09.612973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59961 (local address 127.0.0.1:42723)
0809 00:58:09.613978 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:09.613992 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:09.614043 (+ 51us) server_negotiation.cc:407] Connection header received
0809 00:58:09.615886 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:09.615916 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:09.616409 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:09.616901 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:09.617611 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:09.618988 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:09.620578 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:09.621427 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:09.621680 (+ 253us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:09.623171 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:09.623180 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:09.623198 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:09.623812 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:09.623939 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:09.624002 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:09.624214 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:09.624544 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 00:58:09.624751 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":445,"threads_started":1}
I20260809 00:58:09.628485 10910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1: No bootstrap required, opened a new log
I20260809 00:58:09.630441 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:09.637986 10910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fb30ea1a4a4a85a5925bc51c5315f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } }
I20260809 00:58:09.638574 10910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:09.638715 10910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17fb30ea1a4a4a85a5925bc51c5315f1, State: Initialized, Role: FOLLOWER
I20260809 00:58:09.639482 10910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fb30ea1a4a4a85a5925bc51c5315f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } }
I20260809 00:58:09.639767 10910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:09.639961 10910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:09.640165 10910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:09.643188 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:09.635208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52577 (local address 127.0.0.1:42723)
0809 00:58:09.635463 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:09.635471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:09.635873 (+ 402us) server_negotiation.cc:407] Connection header received
0809 00:58:09.636060 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:09.636067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:09.636160 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:09.636287 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:09.636942 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:09.638095 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:09.639869 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:09.640636 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:09.640781 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:09.642207 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:09.642213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:09.642218 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:09.642244 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:09.642304 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:09.642328 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:09.642474 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:09.642783 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 00:58:09.642848 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 00:58:09.643617 10910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fb30ea1a4a4a85a5925bc51c5315f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } }
I20260809 00:58:09.644318 10910 leader_election.cc:302] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17fb30ea1a4a4a85a5925bc51c5315f1; no voters:
I20260809 00:58:09.645020 10910 leader_election.cc:288] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:09.645147 10923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:09.646059 10923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [term 1 LEADER]: Becoming Leader. State: Replica: 17fb30ea1a4a4a85a5925bc51c5315f1, State: Running, Role: LEADER
I20260809 00:58:09.646941 10923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fb30ea1a4a4a85a5925bc51c5315f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } }
I20260809 00:58:09.647698 10910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:09.650267 10924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17fb30ea1a4a4a85a5925bc51c5315f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fb30ea1a4a4a85a5925bc51c5315f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } } }
I20260809 00:58:09.650801 10924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:09.651731 10925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17fb30ea1a4a4a85a5925bc51c5315f1. Latest consensus state: current_term: 1 leader_uuid: "17fb30ea1a4a4a85a5925bc51c5315f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17fb30ea1a4a4a85a5925bc51c5315f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42723 } } }
I20260809 00:58:09.652143 10925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:09.658260 10930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:09.665210 10930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:09.676473 10930 catalog_manager.cc:1347] Generated new cluster ID: 19adb8af7c3c437193e559ebee699483
I20260809 00:58:09.676631 10930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:09.692174 10930 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:09.694159 10930 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:09.706898 10930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17fb30ea1a4a4a85a5925bc51c5315f1: Generated new TSK 0
I20260809 00:58:09.708189 10930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:09.723523 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 10798
W20260809 00:58:09.724880 10900 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59961
W20260809 00:58:09.725850 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 10798: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:58:09.726048 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 10798
W20260809 00:58:09.761924 10773 connection.cc:535] client connection to 127.0.0.1:42723 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42723): Connection reset by peer (error 104)
2026-08-09T00:58:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:09 dist-test-slave-j2vl krb5kdc[10960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:09 dist-test-slave-j2vl krb5kdc[10960](info): set up 2 sockets
Aug 09 00:58:09 dist-test-slave-j2vl krb5kdc[10960](info): commencing operation
krb5kdc: starting...
W20260809 00:58:11.126667 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.344s user 0.007s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:11 dist-test-slave-j2vl krb5kdc[10960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 00:58:11.499161 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:11.882897 10980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:11.883267 10980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:11.883464 10980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:11.891331 10980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:11.891476 10980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:11.891592 10980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:11.891707 10980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:11.891780 10980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:11.903088 10980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:11.904707 10980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:11.906275 10980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:11.920181 10996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:11.920146 10999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:11.920100 10997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:11.921555 10980 server_base.cc:1034] running on GCE node
I20260809 00:58:11.922565 10980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:11.924310 10980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:11.925565 10980 hybrid_clock.cc:648] HybridClock initialized: now 1786237091925522 us; error 44 us; skew 500 ppm
I20260809 00:58:11.925980 10980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:11 dist-test-slave-j2vl krb5kdc[10960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:11.929689 10980 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 00:58:11.931586 10980 webserver.cc:460] Webserver started at http://127.0.0.1:39861/ using document root <none> and password file <none>
I20260809 00:58:11.932435 10980 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:11.932605 10980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:11.933177 10980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:11.936992 10980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "62f4aba018b044daa1b9b47b03088f8f"
format_stamp: "Formatted at 2026-08-09 00:58:11 on dist-test-slave-j2vl"
server_key: "6e989a980c60e2d6174d91f994e2b320"
server_key_iv: "513cd50ebe98e76087faae6256b6eccb"
server_key_version: "encryptionkey@0"
I20260809 00:58:11.938117 10980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "62f4aba018b044daa1b9b47b03088f8f"
format_stamp: "Formatted at 2026-08-09 00:58:11 on dist-test-slave-j2vl"
server_key: "6e989a980c60e2d6174d91f994e2b320"
server_key_iv: "513cd50ebe98e76087faae6256b6eccb"
server_key_version: "encryptionkey@0"
I20260809 00:58:11.944463 10980 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 00:58:11.948801 11011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:11.950665 10980 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:58:11.950927 10980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "62f4aba018b044daa1b9b47b03088f8f"
format_stamp: "Formatted at 2026-08-09 00:58:11 on dist-test-slave-j2vl"
server_key: "6e989a980c60e2d6174d91f994e2b320"
server_key_iv: "513cd50ebe98e76087faae6256b6eccb"
server_key_version: "encryptionkey@0"
I20260809 00:58:11.951193 10980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:11.959164 10980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:11.964298 10980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:11.964670 10980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:11.985792 10980 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38699
I20260809 00:58:11.985845 11103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38699 every 8 connection(s)
I20260809 00:58:11.987459 10980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:11.991984 11104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:11.997401 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 10980
I20260809 00:58:11.997627 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:11.998247 10116 external_mini_cluster.cc:1428] Setting key 44b2b0b2264ac8fc3d67bbd3bec8990a
I20260809 00:58:12.005913 11104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f: Bootstrap starting.
I20260809 00:58:12.008204 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:12.000511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49707 (local address 127.0.0.1:38699)
0809 00:58:12.001637 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:12.001651 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:12.001699 (+ 48us) server_negotiation.cc:407] Connection header received
0809 00:58:12.004173 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:12.004207 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:12.004701 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:12.005178 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:12.006407 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:12.006414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:12.006430 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:12.006447 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:12.006517 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:12.006542 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:12.006802 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:12.006961 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 00:58:12.007174 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":412,"threads_started":1}
I20260809 00:58:12.009773 11104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:12.011058 11104 log.cc:824] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:12.012470 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:12.014508 11104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f: No bootstrap required, opened a new log
I20260809 00:58:12.020391 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:12.017437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47243 (local address 127.0.0.1:38699)
0809 00:58:12.017686 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:12.017692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:12.018030 (+ 338us) server_negotiation.cc:407] Connection header received
0809 00:58:12.018274 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:12.018278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:12.018351 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:12.018461 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:12.019565 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:12.019570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:12.019574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:12.019591 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:12.019641 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:12.019661 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:12.019774 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:12.019920 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 00:58:12.019981 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260809 00:58:12.024307 11104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f4aba018b044daa1b9b47b03088f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } }
I20260809 00:58:12.024777 11104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:12.024902 11104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f4aba018b044daa1b9b47b03088f8f, State: Initialized, Role: FOLLOWER
I20260809 00:58:12.025657 11104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f4aba018b044daa1b9b47b03088f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } }
I20260809 00:58:12.025914 11104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:12.026103 11104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:12.026310 11104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:12.029158 11104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f4aba018b044daa1b9b47b03088f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } }
I20260809 00:58:12.029856 11104 leader_election.cc:302] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62f4aba018b044daa1b9b47b03088f8f; no voters:
I20260809 00:58:12.030491 11104 leader_election.cc:288] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:12.030689 11118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:12.031443 11118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [term 1 LEADER]: Becoming Leader. State: Replica: 62f4aba018b044daa1b9b47b03088f8f, State: Running, Role: LEADER
I20260809 00:58:12.032137 11118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f4aba018b044daa1b9b47b03088f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } }
I20260809 00:58:12.032920 11104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:12.035563 11120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62f4aba018b044daa1b9b47b03088f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f4aba018b044daa1b9b47b03088f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } } }
I20260809 00:58:12.036015 11120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:12.036592 11121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62f4aba018b044daa1b9b47b03088f8f. Latest consensus state: current_term: 1 leader_uuid: "62f4aba018b044daa1b9b47b03088f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f4aba018b044daa1b9b47b03088f8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38699 } } }
I20260809 00:58:12.036937 11121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:12.037103 11131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:12.046540 11131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:12.059099 11131 catalog_manager.cc:1347] Generated new cluster ID: 414bb468e0cf45dc91aac021d623e268
I20260809 00:58:12.059302 11131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:12.078532 11131 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:12.079094 11131 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:12.091743 11131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62f4aba018b044daa1b9b47b03088f8f: Generated new TSK 0
I20260809 00:58:12.092835 11131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:12.100461 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 10980
W20260809 00:58:12.101835 11094 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49707
W20260809 00:58:12.102681 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 10980: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:58:12.102887 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 10980
2026-08-09T00:58:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:12 dist-test-slave-j2vl krb5kdc[11155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:12 dist-test-slave-j2vl krb5kdc[11155](info): set up 2 sockets
Aug 09 00:58:12 dist-test-slave-j2vl krb5kdc[11155](info): commencing operation
krb5kdc: starting...
W20260809 00:58:13.539304 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.383s user 0.003s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:13 dist-test-slave-j2vl krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 00:58:13.919094 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:14.319681 11181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:14.320045 11181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:14.320218 11181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:14.328045 11181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:14.328207 11181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:14.328287 11181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:14.328387 11181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:14.328459 11181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:14.339304 11181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:14.340819 11181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:14.342344 11181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:14.356254 11195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:14.356361 11187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:14.356357 11188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:14.357636 11181 server_base.cc:1034] running on GCE node
I20260809 00:58:14.358615 11181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:14.360672 11181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:14.361972 11181 hybrid_clock.cc:648] HybridClock initialized: now 1786237094361905 us; error 60 us; skew 500 ppm
I20260809 00:58:14.362437 11181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:14 dist-test-slave-j2vl krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:14.366343 11181 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 00:58:14.368556 11181 webserver.cc:460] Webserver started at http://127.0.0.1:35207/ using document root <none> and password file <none>
I20260809 00:58:14.369518 11181 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:14.369691 11181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:14.370082 11181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:14.373695 11181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "3d0b971708844f76bc53e7897e291cdb"
format_stamp: "Formatted at 2026-08-09 00:58:14 on dist-test-slave-j2vl"
server_key: "cc74150546a916065ee2acb233d39432"
server_key_iv: "1a0c7c264478762eb08e5581b9b07e9d"
server_key_version: "encryptionkey@0"
I20260809 00:58:14.374629 11181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "3d0b971708844f76bc53e7897e291cdb"
format_stamp: "Formatted at 2026-08-09 00:58:14 on dist-test-slave-j2vl"
server_key: "cc74150546a916065ee2acb233d39432"
server_key_iv: "1a0c7c264478762eb08e5581b9b07e9d"
server_key_version: "encryptionkey@0"
I20260809 00:58:14.380944 11181 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260809 00:58:14.385164 11207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:14.386917 11181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 00:58:14.387164 11181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "3d0b971708844f76bc53e7897e291cdb"
format_stamp: "Formatted at 2026-08-09 00:58:14 on dist-test-slave-j2vl"
server_key: "cc74150546a916065ee2acb233d39432"
server_key_iv: "1a0c7c264478762eb08e5581b9b07e9d"
server_key_version: "encryptionkey@0"
I20260809 00:58:14.387419 11181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:14.406828 11181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:14.412014 11181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:14.412366 11181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:14.433810 11181 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40139
I20260809 00:58:14.433856 11289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40139 every 8 connection(s)
I20260809 00:58:14.435395 11181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:14.439849 11291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:14.441824 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 11181
I20260809 00:58:14.442037 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:14.442521 10116 external_mini_cluster.cc:1428] Setting key e65e3f2f6c833c2c74c8869819f9be18
I20260809 00:58:14.453884 11291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb: Bootstrap starting.
I20260809 00:58:14.457713 11291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb: Neither blocks nor log segments found. Creating new log.
Aug 09 00:58:14 dist-test-slave-j2vl krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 00:58:14.459192 11291 log.cc:824] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:14.463719 11291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb: No bootstrap required, opened a new log
I20260809 00:58:14.472671 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:14.444686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46329 (local address 127.0.0.1:40139)
0809 00:58:14.445717 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:14.445730 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:14.445781 (+ 51us) server_negotiation.cc:407] Connection header received
0809 00:58:14.447643 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:14.447674 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:14.448180 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:14.448588 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:14.450901 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:14.452437 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:14.454420 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:14.455187 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:14.455423 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:14.460638 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:14.460651 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:14.460671 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:14.460724 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:14.465303 (+ 4579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:14.466830 (+ 1527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:14.466850 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:14.466862 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:14.467132 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:14.467649 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:14.467654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:14.467657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:14.468547 (+ 890us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:14.468899 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:14.470431 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0809 00:58:14.471001 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":448,"threads_started":1}
I20260809 00:58:14.475764 11291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b971708844f76bc53e7897e291cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40139 } }
I20260809 00:58:14.476410 11291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:14.476549 11291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0b971708844f76bc53e7897e291cdb, State: Initialized, Role: FOLLOWER
I20260809 00:58:14.477324 11291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b971708844f76bc53e7897e291cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40139 } }
I20260809 00:58:14.477718 11291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:14.477914 11291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:14.478116 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:14.478137 11291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:14.481653 11291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b971708844f76bc53e7897e291cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40139 } }
I20260809 00:58:14.482900 11291 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0b971708844f76bc53e7897e291cdb; no voters:
I20260809 00:58:14.484035 11291 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:14.484170 11311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:14.487475 11311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [term 1 LEADER]: Becoming Leader. State: Replica: 3d0b971708844f76bc53e7897e291cdb, State: Running, Role: LEADER
I20260809 00:58:14.489203 11311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b971708844f76bc53e7897e291cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40139 } }
I20260809 00:58:14.491798 11291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:14.495231 11315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d0b971708844f76bc53e7897e291cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b971708844f76bc53e7897e291cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40139 } } }
I20260809 00:58:14.496134 11315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:14.500811 11314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0b971708844f76bc53e7897e291cdb. Latest consensus state: current_term: 1 leader_uuid: "3d0b971708844f76bc53e7897e291cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b971708844f76bc53e7897e291cdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40139 } } }
I20260809 00:58:14.501997 11314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:14.503577 11319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:14.509473 11319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:14.511621 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:14.488099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54923 (local address 127.0.0.1:40139)
0809 00:58:14.489646 (+ 1547us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:14.489655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:14.489685 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:58:14.489780 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:14.489784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:14.489877 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:14.490022 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:14.492163 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:14.493720 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:14.496255 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:14.497023 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:14.497221 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:14.501033 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:14.501045 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:14.501052 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:14.501121 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:14.505576 (+ 4455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:14.507022 (+ 1446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:14.507030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:14.507041 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:14.507345 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:14.508058 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:14.508066 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:14.508076 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:14.508705 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:14.510417 (+ 1712us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:14.510663 (+ 246us) server_negotiation.cc:299] Negotiation successful
0809 00:58:14.511045 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1306}
I20260809 00:58:14.525039 11319 catalog_manager.cc:1347] Generated new cluster ID: 8af61c046b8d42a3a49b50617e7200a7
I20260809 00:58:14.525197 11319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:14.559561 11319 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:14.561070 11319 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:14.569053 11319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d0b971708844f76bc53e7897e291cdb: Generated new TSK 0
I20260809 00:58:14.570379 11319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:14.591041 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11181
I20260809 00:58:14.592557 11281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:14.693099 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 11181
2026-08-09T00:58:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2586 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:14 dist-test-slave-j2vl krb5kdc[11345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:14 dist-test-slave-j2vl krb5kdc[11345](info): set up 2 sockets
Aug 09 00:58:14 dist-test-slave-j2vl krb5kdc[11345](info): commencing operation
krb5kdc: starting...
W20260809 00:58:16.099140 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.356s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:16 dist-test-slave-j2vl krb5kdc[11345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 00:58:16.506235 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:16.896986 11369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:16.897447 11369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:16.897655 11369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:16.905287 11369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:16.905468 11369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:16.905598 11369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:16.905714 11369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:16.905807 11369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:16.916473 11369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:16.918049 11369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:16.919543 11369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:16.932725 11376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:16.932752 11377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:16.932682 11392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:16.933720 11369 server_base.cc:1034] running on GCE node
I20260809 00:58:16.934734 11369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:16.936335 11369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:16.937580 11369 hybrid_clock.cc:648] HybridClock initialized: now 1786237096937513 us; error 55 us; skew 500 ppm
I20260809 00:58:16.938017 11369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:16 dist-test-slave-j2vl krb5kdc[11345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:16.941708 11369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 00:58:16.943495 11369 webserver.cc:460] Webserver started at http://127.0.0.1:33963/ using document root <none> and password file <none>
I20260809 00:58:16.944319 11369 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:16.944482 11369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:16.944836 11369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:16.948396 11369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "6a30eda6eb63433ba78526446af1f6f9"
format_stamp: "Formatted at 2026-08-09 00:58:16 on dist-test-slave-j2vl"
server_key: "621b161e4149b7a45ecbb67861d7cfdb"
server_key_iv: "ca303186fcaa9996bd950e6771a94065"
server_key_version: "encryptionkey@0"
I20260809 00:58:16.949433 11369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "6a30eda6eb63433ba78526446af1f6f9"
format_stamp: "Formatted at 2026-08-09 00:58:16 on dist-test-slave-j2vl"
server_key: "621b161e4149b7a45ecbb67861d7cfdb"
server_key_iv: "ca303186fcaa9996bd950e6771a94065"
server_key_version: "encryptionkey@0"
I20260809 00:58:16.955859 11369 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:58:16.960178 11403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:16.962004 11369 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:58:16.962281 11369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "6a30eda6eb63433ba78526446af1f6f9"
format_stamp: "Formatted at 2026-08-09 00:58:16 on dist-test-slave-j2vl"
server_key: "621b161e4149b7a45ecbb67861d7cfdb"
server_key_iv: "ca303186fcaa9996bd950e6771a94065"
server_key_version: "encryptionkey@0"
I20260809 00:58:16.962560 11369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:16.985370 11369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:16.990625 11369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:16.990998 11369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:17.012864 11369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34681
I20260809 00:58:17.012903 11499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34681 every 8 connection(s)
I20260809 00:58:17.014549 11369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:17.015916 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 11369
I20260809 00:58:17.016130 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:17.016788 10116 external_mini_cluster.cc:1428] Setting key 48313c346b639d8e74e19c524bfde5f1
I20260809 00:58:17.020210 11500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:17.031633 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:17.018923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35499 (local address 127.0.0.1:34681)
0809 00:58:17.019814 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:17.019828 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:17.019877 (+ 49us) server_negotiation.cc:407] Connection header received
0809 00:58:17.021892 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:17.021921 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:17.022397 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:17.022863 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:17.023539 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:17.024967 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:17.026516 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:17.027302 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:17.027560 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:17.028898 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:17.028907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:17.028926 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:17.029649 (+ 723us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:17.029740 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:17.029781 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:17.029932 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:17.030199 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 00:58:17.030519 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":388,"threads_started":1}
I20260809 00:58:17.033070 11500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9: Bootstrap starting.
I20260809 00:58:17.036176 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:17.037043 11500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:17.038313 11500 log.cc:824] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:17.042464 11500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9: No bootstrap required, opened a new log
I20260809 00:58:17.049253 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:17.041225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38789 (local address 127.0.0.1:34681)
0809 00:58:17.041501 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:17.041511 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:17.042021 (+ 510us) server_negotiation.cc:407] Connection header received
0809 00:58:17.042300 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:17.042304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:17.042377 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:17.042476 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:17.043230 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:17.044006 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:17.046090 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:17.046602 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:17.046716 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:17.048347 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:17.048353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:17.048356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:17.048380 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:17.048434 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:17.048461 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:17.048567 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:17.048734 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 00:58:17.048833 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 00:58:17.053334 11500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a30eda6eb63433ba78526446af1f6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34681 } }
I20260809 00:58:17.053937 11500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:17.054052 11500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a30eda6eb63433ba78526446af1f6f9, State: Initialized, Role: FOLLOWER
I20260809 00:58:17.054840 11500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a30eda6eb63433ba78526446af1f6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34681 } }
I20260809 00:58:17.055160 11500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:17.055354 11500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:17.055586 11500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:17.058459 11500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a30eda6eb63433ba78526446af1f6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34681 } }
I20260809 00:58:17.059253 11500 leader_election.cc:302] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a30eda6eb63433ba78526446af1f6f9; no voters:
I20260809 00:58:17.059957 11500 leader_election.cc:288] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:17.060133 11517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:17.060868 11517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a30eda6eb63433ba78526446af1f6f9, State: Running, Role: LEADER
I20260809 00:58:17.061640 11517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a30eda6eb63433ba78526446af1f6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34681 } }
I20260809 00:58:17.062502 11500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:17.065146 11520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a30eda6eb63433ba78526446af1f6f9. Latest consensus state: current_term: 1 leader_uuid: "6a30eda6eb63433ba78526446af1f6f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a30eda6eb63433ba78526446af1f6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34681 } } }
I20260809 00:58:17.065629 11519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a30eda6eb63433ba78526446af1f6f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a30eda6eb63433ba78526446af1f6f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34681 } } }
I20260809 00:58:17.065938 11520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:17.065959 11519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:17.078693 11531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:17.083999 11531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:17.093595 11531 catalog_manager.cc:1347] Generated new cluster ID: 18816a3bdc844f9e9f1f2cd0862776ea
I20260809 00:58:17.093747 11531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:17.111277 11531 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:17.112885 11531 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:17.121627 11531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a30eda6eb63433ba78526446af1f6f9: Generated new TSK 0
I20260809 00:58:17.122789 11531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:17.139678 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11369
W20260809 00:58:17.141597 11495 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35499
W20260809 00:58:17.142484 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11369: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:58:17.142647 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 11369
2026-08-09T00:58:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:17 dist-test-slave-j2vl krb5kdc[11551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:17 dist-test-slave-j2vl krb5kdc[11551](info): set up 2 sockets
Aug 09 00:58:17 dist-test-slave-j2vl krb5kdc[11551](info): commencing operation
krb5kdc: starting...
W20260809 00:58:18.554556 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.360s user 0.001s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:18 dist-test-slave-j2vl krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 00:58:18.937094 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:19.339094 11573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:19.339540 11573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:19.339717 11573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:19.347543 11573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:19.347759 11573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:19.347886 11573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:19.348013 11573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:19.348106 11573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:19.359416 11573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:19.361377 11573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:19.363303 11573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:19.382292 11580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:19.382258 11579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:19.382231 11583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:19.384871 11573 server_base.cc:1034] running on GCE node
I20260809 00:58:19.386114 11573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:19.388108 11573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:19.389482 11573 hybrid_clock.cc:648] HybridClock initialized: now 1786237099389396 us; error 89 us; skew 500 ppm
I20260809 00:58:19.389950 11573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:19 dist-test-slave-j2vl krb5kdc[11551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:19.394181 11573 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 00:58:19.396301 11573 webserver.cc:460] Webserver started at http://127.0.0.1:42021/ using document root <none> and password file <none>
I20260809 00:58:19.397186 11573 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:19.397426 11573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:19.397820 11573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:19.401432 11573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "013d6120985941d2ab06a2bc782499b2"
format_stamp: "Formatted at 2026-08-09 00:58:19 on dist-test-slave-j2vl"
server_key: "3c11c548be06cb532a0d82a0ef7e7602"
server_key_iv: "53b573540873649a4566091990435b59"
server_key_version: "encryptionkey@0"
I20260809 00:58:19.402418 11573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "013d6120985941d2ab06a2bc782499b2"
format_stamp: "Formatted at 2026-08-09 00:58:19 on dist-test-slave-j2vl"
server_key: "3c11c548be06cb532a0d82a0ef7e7602"
server_key_iv: "53b573540873649a4566091990435b59"
server_key_version: "encryptionkey@0"
I20260809 00:58:19.409139 11573 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260809 00:58:19.413627 11601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:19.415380 11573 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:58:19.415650 11573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "013d6120985941d2ab06a2bc782499b2"
format_stamp: "Formatted at 2026-08-09 00:58:19 on dist-test-slave-j2vl"
server_key: "3c11c548be06cb532a0d82a0ef7e7602"
server_key_iv: "53b573540873649a4566091990435b59"
server_key_version: "encryptionkey@0"
I20260809 00:58:19.415917 11573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:19.422905 11573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:19.427793 11573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:19.428157 11573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:19.450977 11573 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38451
I20260809 00:58:19.451051 11680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38451 every 8 connection(s)
I20260809 00:58:19.452754 11573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:19.457607 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 11573
I20260809 00:58:19.457835 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:19.457762 11681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:19.458609 10116 external_mini_cluster.cc:1428] Setting key 163bef62942ce1790027a88ac5545c28
I20260809 00:58:19.470455 11685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:19.461743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39611 (local address 127.0.0.1:38451)
0809 00:58:19.463343 (+ 1600us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:19.463360 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:19.463412 (+ 52us) server_negotiation.cc:407] Connection header received
0809 00:58:19.465938 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:19.465977 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:19.466530 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:19.466951 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:19.468444 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:19.468451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:19.468469 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:19.468490 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:19.468572 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:19.468600 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:19.468865 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:19.468996 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 00:58:19.469272 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1272,"thread_start_us":815,"threads_started":1}
I20260809 00:58:19.475960 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:19.476596 11681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2: Bootstrap starting.
I20260809 00:58:19.480772 11681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:19.482551 11681 log.cc:824] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:19.484522 11685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:19.481500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42485 (local address 127.0.0.1:38451)
0809 00:58:19.481714 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:19.481720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:19.482192 (+ 472us) server_negotiation.cc:407] Connection header received
0809 00:58:19.482396 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:19.482402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:19.482476 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:19.482624 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:19.483755 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:19.483764 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:19.483770 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:19.483793 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:19.483846 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:19.483863 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:19.483999 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:19.484142 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 00:58:19.484216 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 00:58:19.486402 11681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2: No bootstrap required, opened a new log
I20260809 00:58:19.495102 11681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013d6120985941d2ab06a2bc782499b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } }
I20260809 00:58:19.495575 11681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:19.495648 11681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013d6120985941d2ab06a2bc782499b2, State: Initialized, Role: FOLLOWER
I20260809 00:58:19.496362 11681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013d6120985941d2ab06a2bc782499b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } }
I20260809 00:58:19.496608 11681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:19.496778 11681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:19.496981 11681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:19.499866 11681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013d6120985941d2ab06a2bc782499b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } }
I20260809 00:58:19.500523 11681 leader_election.cc:302] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 013d6120985941d2ab06a2bc782499b2; no voters:
I20260809 00:58:19.501214 11681 leader_election.cc:288] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:19.501485 11698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:19.502252 11698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [term 1 LEADER]: Becoming Leader. State: Replica: 013d6120985941d2ab06a2bc782499b2, State: Running, Role: LEADER
I20260809 00:58:19.502936 11698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013d6120985941d2ab06a2bc782499b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } }
I20260809 00:58:19.503688 11681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:19.506789 11700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 013d6120985941d2ab06a2bc782499b2. Latest consensus state: current_term: 1 leader_uuid: "013d6120985941d2ab06a2bc782499b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013d6120985941d2ab06a2bc782499b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } } }
I20260809 00:58:19.507287 11699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "013d6120985941d2ab06a2bc782499b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013d6120985941d2ab06a2bc782499b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38451 } } }
I20260809 00:58:19.508818 11700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:19.509275 11699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:19.515102 11711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:19.520558 11711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:19.531417 11711 catalog_manager.cc:1347] Generated new cluster ID: bae36ed94de64e6e81563322d2f7b884
I20260809 00:58:19.531574 11711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:19.543895 11711 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:19.544564 11711 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:19.558236 11711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 013d6120985941d2ab06a2bc782499b2: Generated new TSK 0
I20260809 00:58:19.559314 11711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:19.576325 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11573
W20260809 00:58:19.577637 11675 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39611
W20260809 00:58:19.578555 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11573: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:58:19.578728 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 11573
2026-08-09T00:58:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:19 dist-test-slave-j2vl krb5kdc[11731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:19 dist-test-slave-j2vl krb5kdc[11731](info): set up 2 sockets
Aug 09 00:58:19 dist-test-slave-j2vl krb5kdc[11731](info): commencing operation
krb5kdc: starting...
W20260809 00:58:21.015465 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.381s user 0.004s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:21 dist-test-slave-j2vl krb5kdc[11731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 00:58:21.390116 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:21.783151 11751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:21.783525 11751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:21.783708 11751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:21.791714 11751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:21.791885 11751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:21.791987 11751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:21.792105 11751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:21.792168 11751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:21.803432 11751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:21.805054 11751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:21.806628 11751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:21.820649 11762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:21.820619 11769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:21.820585 11765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:21.821550 11751 server_base.cc:1034] running on GCE node
I20260809 00:58:21.822587 11751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:21.824258 11751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:21.825484 11751 hybrid_clock.cc:648] HybridClock initialized: now 1786237101825438 us; error 36 us; skew 500 ppm
I20260809 00:58:21.825913 11751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:21 dist-test-slave-j2vl krb5kdc[11731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:21.829305 11751 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 00:58:21.831188 11751 webserver.cc:460] Webserver started at http://127.0.0.1:37399/ using document root <none> and password file <none>
I20260809 00:58:21.832016 11751 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:21.832182 11751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:21.832548 11751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:21.836074 11751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "6d10db2ad9284992af0376584bc54be2"
format_stamp: "Formatted at 2026-08-09 00:58:21 on dist-test-slave-j2vl"
server_key: "3a9210fb2f405a7565564045cab72c43"
server_key_iv: "7d9d157c90904cd6c207c6b59146749a"
server_key_version: "encryptionkey@0"
I20260809 00:58:21.836966 11751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "6d10db2ad9284992af0376584bc54be2"
format_stamp: "Formatted at 2026-08-09 00:58:21 on dist-test-slave-j2vl"
server_key: "3a9210fb2f405a7565564045cab72c43"
server_key_iv: "7d9d157c90904cd6c207c6b59146749a"
server_key_version: "encryptionkey@0"
I20260809 00:58:21.843295 11751 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260809 00:58:21.847612 11785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:21.849310 11751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 00:58:21.849654 11751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "6d10db2ad9284992af0376584bc54be2"
format_stamp: "Formatted at 2026-08-09 00:58:21 on dist-test-slave-j2vl"
server_key: "3a9210fb2f405a7565564045cab72c43"
server_key_iv: "7d9d157c90904cd6c207c6b59146749a"
server_key_version: "encryptionkey@0"
I20260809 00:58:21.849934 11751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:21.868520 11751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:21.873991 11751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:21.874351 11751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:21.895426 11751 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40817
I20260809 00:58:21.895541 11868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40817 every 8 connection(s)
I20260809 00:58:21.897095 11751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:21.898048 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 11751
I20260809 00:58:21.898275 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:21.898888 10116 external_mini_cluster.cc:1428] Setting key 10b83ad1056a705f4f7c6a6fe09d0669
I20260809 00:58:21.902357 11869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 00:58:21 dist-test-slave-j2vl krb5kdc[11731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 00:58:21.917908 11869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2: Bootstrap starting.
I20260809 00:58:21.922817 11869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:21.924364 11869 log.cc:824] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:21.927256 11874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:21.901236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.139:53489 (local address 127.0.0.1:40817)
0809 00:58:21.902185 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:21.902197 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:21.902241 (+ 44us) server_negotiation.cc:407] Connection header received
0809 00:58:21.904037 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:21.904063 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:21.904506 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:21.904875 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:21.906984 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:21.908482 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:21.910414 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:21.911184 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:21.911343 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:21.917113 (+ 5770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:21.917124 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:21.917142 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:21.917195 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:21.921610 (+ 4415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:21.922780 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:21.922797 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:21.922811 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:21.923053 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:21.923571 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:21.923576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:21.923579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:21.924407 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:21.924752 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:21.925438 (+ 686us) server_negotiation.cc:299] Negotiation successful
0809 00:58:21.925831 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":400,"threads_started":1}
I20260809 00:58:21.928336 11869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2: No bootstrap required, opened a new log
I20260809 00:58:21.931844 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:21.937609 11869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d10db2ad9284992af0376584bc54be2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40817 } }
I20260809 00:58:21.938251 11869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:21.938366 11869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d10db2ad9284992af0376584bc54be2, State: Initialized, Role: FOLLOWER
I20260809 00:58:21.939152 11869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d10db2ad9284992af0376584bc54be2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40817 } }
I20260809 00:58:21.939419 11869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:21.939579 11869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:21.939814 11869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:21.942924 11869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d10db2ad9284992af0376584bc54be2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40817 } }
I20260809 00:58:21.943661 11869 leader_election.cc:302] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d10db2ad9284992af0376584bc54be2; no voters:
I20260809 00:58:21.944476 11869 leader_election.cc:288] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:21.944612 11892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:21.947049 11869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:21.948079 11892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [term 1 LEADER]: Becoming Leader. State: Replica: 6d10db2ad9284992af0376584bc54be2, State: Running, Role: LEADER
I20260809 00:58:21.948781 11892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d10db2ad9284992af0376584bc54be2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40817 } }
I20260809 00:58:21.952406 11894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d10db2ad9284992af0376584bc54be2. Latest consensus state: current_term: 1 leader_uuid: "6d10db2ad9284992af0376584bc54be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d10db2ad9284992af0376584bc54be2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40817 } } }
I20260809 00:58:21.953430 11894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:21.953161 11893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d10db2ad9284992af0376584bc54be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d10db2ad9284992af0376584bc54be2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40817 } } }
I20260809 00:58:21.953725 11893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:21.955420 11907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:21.959410 11874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:21.936981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.139:49835 (local address 127.0.0.1:40817)
0809 00:58:21.937167 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:21.937172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:21.937726 (+ 554us) server_negotiation.cc:407] Connection header received
0809 00:58:21.937880 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:21.937885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:21.937979 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:21.938105 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:21.940115 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:21.941408 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:21.942719 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:21.943408 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:21.943543 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:21.947805 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:21.947812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:21.947817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:21.947875 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:21.953253 (+ 5378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:21.955031 (+ 1778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:21.955037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:21.955040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:21.955285 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:21.955997 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:21.956002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:21.956004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:21.956459 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:21.958134 (+ 1675us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:21.958408 (+ 274us) server_negotiation.cc:299] Negotiation successful
0809 00:58:21.958835 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 00:58:21.965102 11907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:21.976972 11907 catalog_manager.cc:1347] Generated new cluster ID: fe5038de0e6540fc841f89914f7e12ce
I20260809 00:58:21.977133 11907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:21.991595 11907 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:21.993125 11907 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:22.003448 11907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d10db2ad9284992af0376584bc54be2: Generated new TSK 0
I20260809 00:58:22.004498 11907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:22.010640 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11751
I20260809 00:58:22.011946 11860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:22.108733 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 11751
W20260809 00:58:22.140723 11728 connection.cc:535] client connection to 127.0.0.1:40817 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40817): Connection reset by peer (error 104)
2026-08-09T00:58:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2526 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:22 dist-test-slave-j2vl krb5kdc[11935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:22 dist-test-slave-j2vl krb5kdc[11935](info): set up 2 sockets
Aug 09 00:58:22 dist-test-slave-j2vl krb5kdc[11935](info): commencing operation
krb5kdc: starting...
W20260809 00:58:23.509884 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.349s user 0.008s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:23 dist-test-slave-j2vl krb5kdc[11935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 00:58:23.895429 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:24.292914 11956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:24.293275 11956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:24.293484 11956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:24.301222 11956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:24.301429 11956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:24.301534 11956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:24.301628 11956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:24.301714 11956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:24.312731 11956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:24.314287 11956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:24.315786 11956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:24.329044 11972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:24.329151 11969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:24.329432 11968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:24.330310 11956 server_base.cc:1034] running on GCE node
I20260809 00:58:24.331259 11956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:24.332971 11956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:24.334251 11956 hybrid_clock.cc:648] HybridClock initialized: now 1786237104334179 us; error 66 us; skew 500 ppm
I20260809 00:58:24.334743 11956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:24 dist-test-slave-j2vl krb5kdc[11935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:24.338497 11956 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 00:58:24.340281 11956 webserver.cc:460] Webserver started at http://127.0.0.1:39773/ using document root <none> and password file <none>
I20260809 00:58:24.341114 11956 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:24.341281 11956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:24.341656 11956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:24.345212 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "691f40b7bd8c4c14b80703755430f008"
format_stamp: "Formatted at 2026-08-09 00:58:24 on dist-test-slave-j2vl"
server_key: "71813f0aa9bb24844f4282c7fa52a87e"
server_key_iv: "aa40c54eb92e763cde54c8eee1c72542"
server_key_version: "encryptionkey@0"
I20260809 00:58:24.346167 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "691f40b7bd8c4c14b80703755430f008"
format_stamp: "Formatted at 2026-08-09 00:58:24 on dist-test-slave-j2vl"
server_key: "71813f0aa9bb24844f4282c7fa52a87e"
server_key_iv: "aa40c54eb92e763cde54c8eee1c72542"
server_key_version: "encryptionkey@0"
I20260809 00:58:24.352377 11956 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260809 00:58:24.356845 11987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:24.358863 11956 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 00:58:24.359153 11956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "691f40b7bd8c4c14b80703755430f008"
format_stamp: "Formatted at 2026-08-09 00:58:24 on dist-test-slave-j2vl"
server_key: "71813f0aa9bb24844f4282c7fa52a87e"
server_key_iv: "aa40c54eb92e763cde54c8eee1c72542"
server_key_version: "encryptionkey@0"
I20260809 00:58:24.359407 11956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:24.380422 11956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:24.385561 11956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:24.385847 11956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:24.407287 11956 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33571
I20260809 00:58:24.407402 12073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33571 every 8 connection(s)
I20260809 00:58:24.408893 11956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:24.413398 12074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:24.414935 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 11956
I20260809 00:58:24.415133 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:24.415637 10116 external_mini_cluster.cc:1428] Setting key 5bab152083910eae6568a8edd0788254
I20260809 00:58:24.431188 12074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008: Bootstrap starting.
I20260809 00:58:24.432798 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:24.417824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.139:46141 (local address 127.0.0.1:33571)
0809 00:58:24.418751 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:24.418768 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:24.418828 (+ 60us) server_negotiation.cc:407] Connection header received
0809 00:58:24.420900 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:24.420942 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:24.421661 (+ 719us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:24.422236 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:24.422835 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:24.424848 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:24.426711 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:24.428006 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:24.428238 (+ 232us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:24.429635 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:24.429648 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:24.429671 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:24.430505 (+ 834us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:24.430640 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:24.430681 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:24.430873 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:24.431151 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 00:58:24.431367 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":386,"threads_started":1}
I20260809 00:58:24.435370 12074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:24.436764 12074 log.cc:824] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:24.437480 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:24.440027 12074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008: No bootstrap required, opened a new log
I20260809 00:58:24.449028 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:24.442199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.139:59427 (local address 127.0.0.1:33571)
0809 00:58:24.442407 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:24.442421 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:24.442893 (+ 472us) server_negotiation.cc:407] Connection header received
0809 00:58:24.443141 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:24.443146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:24.443222 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:24.443359 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:24.443915 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:24.444710 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:24.446198 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:24.446717 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:24.446911 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:24.448224 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:24.448229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:24.448233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:24.448253 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:24.448300 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:24.448316 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:24.448446 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:24.448657 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 00:58:24.448719 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 00:58:24.449263 12074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691f40b7bd8c4c14b80703755430f008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } }
I20260809 00:58:24.449904 12074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:24.450048 12074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691f40b7bd8c4c14b80703755430f008, State: Initialized, Role: FOLLOWER
I20260809 00:58:24.450739 12074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691f40b7bd8c4c14b80703755430f008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } }
I20260809 00:58:24.451081 12074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:24.451258 12074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:24.451455 12074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:24.453989 12074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691f40b7bd8c4c14b80703755430f008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } }
I20260809 00:58:24.454588 12074 leader_election.cc:302] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 691f40b7bd8c4c14b80703755430f008; no voters:
I20260809 00:58:24.455418 12074 leader_election.cc:288] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:24.455557 12093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:24.456228 12093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [term 1 LEADER]: Becoming Leader. State: Replica: 691f40b7bd8c4c14b80703755430f008, State: Running, Role: LEADER
I20260809 00:58:24.456888 12093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691f40b7bd8c4c14b80703755430f008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } }
I20260809 00:58:24.458192 12074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:24.460525 12094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691f40b7bd8c4c14b80703755430f008. Latest consensus state: current_term: 1 leader_uuid: "691f40b7bd8c4c14b80703755430f008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691f40b7bd8c4c14b80703755430f008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } } }
I20260809 00:58:24.460528 12098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "691f40b7bd8c4c14b80703755430f008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691f40b7bd8c4c14b80703755430f008" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } } }
I20260809 00:58:24.461148 12094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:24.461278 12098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:24.471683 12104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:24.480813 12104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:24.493927 12104 catalog_manager.cc:1347] Generated new cluster ID: cdde59c1691f43a79bc028436970f252
I20260809 00:58:24.494078 12104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:24.506700 12104 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:24.508352 12104 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:24.525417 12104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 691f40b7bd8c4c14b80703755430f008: Generated new TSK 0
I20260809 00:58:24.526690 12104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:24.533006 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11956
W20260809 00:58:24.534580 12068 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.139:46141
W20260809 00:58:24.535533 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 11956: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:58:24.535713 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 11956
W20260809 00:58:24.567354 11928 connection.cc:535] client connection to 127.0.0.1:33571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33571): Connection reset by peer (error 104)
2026-08-09T00:58:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:24 dist-test-slave-j2vl krb5kdc[12124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:24 dist-test-slave-j2vl krb5kdc[12124](info): set up 2 sockets
Aug 09 00:58:24 dist-test-slave-j2vl krb5kdc[12124](info): commencing operation
krb5kdc: starting...
W20260809 00:58:25.933924 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.346s user 0.001s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:26 dist-test-slave-j2vl krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 00:58:26.325753 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:26.722587 12150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:26.723033 12150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:26.723212 12150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:26.731202 12150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:26.731417 12150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:26.731546 12150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:26.731667 12150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:26.731752 12150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:26.742761 12150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:26.744400 12150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:26.745993 12150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:26.760463 12161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:26.760453 12158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:26.761183 12164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:26.761507 12150 server_base.cc:1034] running on GCE node
I20260809 00:58:26.762530 12150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:26.764283 12150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:26.765534 12150 hybrid_clock.cc:648] HybridClock initialized: now 1786237106765464 us; error 74 us; skew 500 ppm
I20260809 00:58:26.765960 12150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:26 dist-test-slave-j2vl krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:26.769557 12150 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 00:58:26.771385 12150 webserver.cc:460] Webserver started at http://127.0.0.1:46335/ using document root <none> and password file <none>
I20260809 00:58:26.772182 12150 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:26.772358 12150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:26.772713 12150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:26.776208 12150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "05600d1c19154f4587fbb5ab48ff8c6c"
format_stamp: "Formatted at 2026-08-09 00:58:26 on dist-test-slave-j2vl"
server_key: "af887e2c4aac737eff2ac7cc90495f21"
server_key_iv: "263b72dbdacf7654dd3b7a1640971221"
server_key_version: "encryptionkey@0"
I20260809 00:58:26.777132 12150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "05600d1c19154f4587fbb5ab48ff8c6c"
format_stamp: "Formatted at 2026-08-09 00:58:26 on dist-test-slave-j2vl"
server_key: "af887e2c4aac737eff2ac7cc90495f21"
server_key_iv: "263b72dbdacf7654dd3b7a1640971221"
server_key_version: "encryptionkey@0"
I20260809 00:58:26.783593 12150 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260809 00:58:26.788090 12183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:26.789923 12150 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260809 00:58:26.790179 12150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "05600d1c19154f4587fbb5ab48ff8c6c"
format_stamp: "Formatted at 2026-08-09 00:58:26 on dist-test-slave-j2vl"
server_key: "af887e2c4aac737eff2ac7cc90495f21"
server_key_iv: "263b72dbdacf7654dd3b7a1640971221"
server_key_version: "encryptionkey@0"
I20260809 00:58:26.790562 12150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:26.798552 12150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:26.806002 12150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:26.806483 12150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:26.829111 12150 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40605
I20260809 00:58:26.829157 12270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40605 every 8 connection(s)
I20260809 00:58:26.830663 12150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:26.834995 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 12150
I20260809 00:58:26.835222 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:26.835712 12271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:26.835825 10116 external_mini_cluster.cc:1428] Setting key 85a2540660865954d500ede6ba63750b
I20260809 00:58:26.845275 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:26.837909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.139:42297 (local address 127.0.0.1:40605)
0809 00:58:26.838814 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:26.838827 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:26.838880 (+ 53us) server_negotiation.cc:407] Connection header received
0809 00:58:26.841158 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:26.841186 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:26.841688 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:26.842169 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:26.843395 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:26.843402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:26.843417 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:26.843432 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:26.843499 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:26.843521 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:26.843803 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:26.843968 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 00:58:26.844212 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":392,"threads_started":1}
I20260809 00:58:26.849988 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:26.851413 12271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c: Bootstrap starting.
I20260809 00:58:26.855836 12271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:26.857241 12271 log.cc:824] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:26.858707 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:26.855340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.139:46043 (local address 127.0.0.1:40605)
0809 00:58:26.855888 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:26.855896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:26.856190 (+ 294us) server_negotiation.cc:407] Connection header received
0809 00:58:26.856439 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:26.856444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:26.856502 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:26.856618 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:26.857960 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:26.857981 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:26.857985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:26.858009 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:26.858063 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:26.858081 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:26.858242 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:26.858336 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 00:58:26.858421 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390}
I20260809 00:58:26.860879 12271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c: No bootstrap required, opened a new log
I20260809 00:58:26.870036 12271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05600d1c19154f4587fbb5ab48ff8c6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } }
I20260809 00:58:26.870517 12271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:26.870631 12271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05600d1c19154f4587fbb5ab48ff8c6c, State: Initialized, Role: FOLLOWER
I20260809 00:58:26.871338 12271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05600d1c19154f4587fbb5ab48ff8c6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } }
I20260809 00:58:26.871582 12271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:26.871793 12271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:26.872020 12271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:26.874868 12271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05600d1c19154f4587fbb5ab48ff8c6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } }
I20260809 00:58:26.875420 12271 leader_election.cc:302] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05600d1c19154f4587fbb5ab48ff8c6c; no voters:
I20260809 00:58:26.876036 12271 leader_election.cc:288] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:26.876245 12287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:26.877054 12287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [term 1 LEADER]: Becoming Leader. State: Replica: 05600d1c19154f4587fbb5ab48ff8c6c, State: Running, Role: LEADER
I20260809 00:58:26.877924 12287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05600d1c19154f4587fbb5ab48ff8c6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } }
I20260809 00:58:26.878710 12271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:26.882190 12289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05600d1c19154f4587fbb5ab48ff8c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05600d1c19154f4587fbb5ab48ff8c6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } } }
I20260809 00:58:26.882154 12290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05600d1c19154f4587fbb5ab48ff8c6c. Latest consensus state: current_term: 1 leader_uuid: "05600d1c19154f4587fbb5ab48ff8c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05600d1c19154f4587fbb5ab48ff8c6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40605 } } }
I20260809 00:58:26.882802 12289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:26.882969 12290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:26.885332 12302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:26.891310 12302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:26.902956 12302 catalog_manager.cc:1347] Generated new cluster ID: 5c82dbf99d08489da99f06ec543b1e4f
I20260809 00:58:26.903108 12302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:26.920135 12302 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:26.920773 12302 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:26.940871 12302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05600d1c19154f4587fbb5ab48ff8c6c: Generated new TSK 0
I20260809 00:58:26.942193 12302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:26.958033 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12150
W20260809 00:58:26.959429 12265 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.139:42297
W20260809 00:58:26.960412 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12150: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:58:26.960716 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 12150
2026-08-09T00:58:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:27 dist-test-slave-j2vl krb5kdc[12319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:27 dist-test-slave-j2vl krb5kdc[12319](info): set up 2 sockets
Aug 09 00:58:27 dist-test-slave-j2vl krb5kdc[12319](info): commencing operation
krb5kdc: starting...
W20260809 00:58:28.369994 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.362s user 0.004s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:28 dist-test-slave-j2vl krb5kdc[12319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 00:58:28.763067 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:29.157694 12344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:29.158131 12344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:29.158316 12344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:29.166275 12344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:29.166476 12344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:29.166601 12344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:29.166716 12344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:29.166811 12344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:29.178594 12344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:29.180387 12344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:29.182106 12344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:29.197285 12351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:29.197208 12352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:29.197256 12357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:29.199239 12344 server_base.cc:1034] running on GCE node
I20260809 00:58:29.200255 12344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:29.202140 12344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:29.203413 12344 hybrid_clock.cc:648] HybridClock initialized: now 1786237109203359 us; error 56 us; skew 500 ppm
I20260809 00:58:29.203835 12344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:29 dist-test-slave-j2vl krb5kdc[12319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:29.207681 12344 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 00:58:29.209717 12344 webserver.cc:460] Webserver started at http://127.0.0.1:35845/ using document root <none> and password file <none>
I20260809 00:58:29.210541 12344 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:29.210711 12344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:29.211078 12344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:29.214695 12344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "215855912a56474dac638cba18dced25"
format_stamp: "Formatted at 2026-08-09 00:58:29 on dist-test-slave-j2vl"
server_key: "4d590f64182f5f137357e49b04f4f682"
server_key_iv: "a6158e17ef0e8c3e5143b0525a88c4af"
server_key_version: "encryptionkey@0"
I20260809 00:58:29.215649 12344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "215855912a56474dac638cba18dced25"
format_stamp: "Formatted at 2026-08-09 00:58:29 on dist-test-slave-j2vl"
server_key: "4d590f64182f5f137357e49b04f4f682"
server_key_iv: "a6158e17ef0e8c3e5143b0525a88c4af"
server_key_version: "encryptionkey@0"
I20260809 00:58:29.222023 12344 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 00:58:29.226361 12365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:29.228091 12344 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:58:29.228346 12344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "215855912a56474dac638cba18dced25"
format_stamp: "Formatted at 2026-08-09 00:58:29 on dist-test-slave-j2vl"
server_key: "4d590f64182f5f137357e49b04f4f682"
server_key_iv: "a6158e17ef0e8c3e5143b0525a88c4af"
server_key_version: "encryptionkey@0"
I20260809 00:58:29.228609 12344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:29.250542 12344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:29.256014 12344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:29.256404 12344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:29.278519 12344 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34857
I20260809 00:58:29.278544 12453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34857 every 8 connection(s)
I20260809 00:58:29.280087 12344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:29.281864 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 12344
I20260809 00:58:29.282058 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:29.282684 10116 external_mini_cluster.cc:1428] Setting key 6773254e32057539597dceb12ededca8
I20260809 00:58:29.285167 12455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 00:58:29 dist-test-slave-j2vl krb5kdc[12319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 00:58:29.301170 12455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25: Bootstrap starting.
I20260809 00:58:29.306140 12455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:29.307780 12455 log.cc:824] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:29.311825 12455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25: No bootstrap required, opened a new log
I20260809 00:58:29.311849 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:29.285044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.139:35729 (local address 127.0.0.1:34857)
0809 00:58:29.286368 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:29.286382 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:29.286438 (+ 56us) server_negotiation.cc:407] Connection header received
0809 00:58:29.288210 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:29.288241 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:29.288769 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:29.289214 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:29.291473 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:29.293056 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:29.295073 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:29.295891 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:29.296049 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:29.301536 (+ 5487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:29.301553 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:29.301583 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:29.301646 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:29.306024 (+ 4378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:29.307222 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:29.307233 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:29.307243 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:29.307530 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:29.308165 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:29.308171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:29.308174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:29.308953 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:29.309280 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:29.309993 (+ 713us) server_negotiation.cc:299] Negotiation successful
0809 00:58:29.310370 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1077,"thread_start_us":770,"threads_started":1}
I20260809 00:58:29.316682 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:29.322760 12455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215855912a56474dac638cba18dced25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34857 } }
I20260809 00:58:29.323339 12455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:29.323490 12455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215855912a56474dac638cba18dced25, State: Initialized, Role: FOLLOWER
I20260809 00:58:29.324337 12455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215855912a56474dac638cba18dced25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34857 } }
I20260809 00:58:29.324648 12455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:29.324820 12455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:29.325042 12455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:29.328208 12455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215855912a56474dac638cba18dced25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34857 } }
I20260809 00:58:29.328889 12455 leader_election.cc:302] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 215855912a56474dac638cba18dced25; no voters:
I20260809 00:58:29.329797 12455 leader_election.cc:288] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:29.329954 12471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:29.332262 12455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:29.333681 12471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [term 1 LEADER]: Becoming Leader. State: Replica: 215855912a56474dac638cba18dced25, State: Running, Role: LEADER
I20260809 00:58:29.334565 12471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215855912a56474dac638cba18dced25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34857 } }
I20260809 00:58:29.338229 12472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "215855912a56474dac638cba18dced25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215855912a56474dac638cba18dced25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34857 } } }
I20260809 00:58:29.338781 12472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:29.341441 12474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 215855912a56474dac638cba18dced25. Latest consensus state: current_term: 1 leader_uuid: "215855912a56474dac638cba18dced25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "215855912a56474dac638cba18dced25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34857 } } }
I20260809 00:58:29.342746 12474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:29.346426 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:29.322947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.139:36857 (local address 127.0.0.1:34857)
0809 00:58:29.323175 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:29.323181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:29.323648 (+ 467us) server_negotiation.cc:407] Connection header received
0809 00:58:29.323840 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:29.323846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:29.323947 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:29.324069 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:29.326087 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:29.327310 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:29.328916 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:29.329768 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:29.329925 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:29.334541 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:29.334553 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:29.334561 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:29.334622 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:29.339940 (+ 5318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:29.341692 (+ 1752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:29.341700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:29.341706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:29.341952 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:29.342814 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:29.342823 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:29.342829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:29.343392 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:29.345218 (+ 1826us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:29.345520 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 00:58:29.345829 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 00:58:29.348579 12484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:29.355036 12484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:29.367530 12484 catalog_manager.cc:1347] Generated new cluster ID: 66acc2809bf34b16baf61d627e62057f
I20260809 00:58:29.367677 12484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:29.386165 12484 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:29.387830 12484 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:29.397998 12484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 215855912a56474dac638cba18dced25: Generated new TSK 0
I20260809 00:58:29.399356 12484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:29.409497 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12344
I20260809 00:58:29.410918 12445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:29.523334 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 12344
W20260809 00:58:29.556341 12316 connection.cc:535] client connection to 127.0.0.1:34857 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34857): Connection reset by peer (error 104)
2026-08-09T00:58:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2568 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:29 dist-test-slave-j2vl krb5kdc[12510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:29 dist-test-slave-j2vl krb5kdc[12510](info): set up 2 sockets
Aug 09 00:58:29 dist-test-slave-j2vl krb5kdc[12510](info): commencing operation
krb5kdc: starting...
W20260809 00:58:30.929289 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.351s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:31 dist-test-slave-j2vl krb5kdc[12510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 00:58:31.308936 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:31.700516 12530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:31.701038 12530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:31.701218 12530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:31.709185 12530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:31.709425 12530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:31.709548 12530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:31.709666 12530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:31.709756 12530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:31.721935 12530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:31.723548 12530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:31.725050 12530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:31.738838 12538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:31.738946 12541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:31.739217 12544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:31.740167 12530 server_base.cc:1034] running on GCE node
I20260809 00:58:31.741192 12530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:31.743007 12530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:31.744248 12530 hybrid_clock.cc:648] HybridClock initialized: now 1786237111744200 us; error 42 us; skew 500 ppm
I20260809 00:58:31.744663 12530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:31 dist-test-slave-j2vl krb5kdc[12510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:31.752509 12530 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 00:58:31.754489 12530 webserver.cc:460] Webserver started at http://127.0.0.1:44663/ using document root <none> and password file <none>
I20260809 00:58:31.755362 12530 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:31.755545 12530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:31.755895 12530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:31.759496 12530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "5a3422f4ddcf41ab8248cff31b65b6b1"
format_stamp: "Formatted at 2026-08-09 00:58:31 on dist-test-slave-j2vl"
server_key: "93e6dd8facc08e2095fc28ae6feb13cd"
server_key_iv: "78e51565bf9b845792476f40e6e8c708"
server_key_version: "encryptionkey@0"
I20260809 00:58:31.760461 12530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "5a3422f4ddcf41ab8248cff31b65b6b1"
format_stamp: "Formatted at 2026-08-09 00:58:31 on dist-test-slave-j2vl"
server_key: "93e6dd8facc08e2095fc28ae6feb13cd"
server_key_iv: "78e51565bf9b845792476f40e6e8c708"
server_key_version: "encryptionkey@0"
I20260809 00:58:31.766985 12530 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 00:58:31.771538 12552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:31.773479 12530 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 00:58:31.773739 12530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "5a3422f4ddcf41ab8248cff31b65b6b1"
format_stamp: "Formatted at 2026-08-09 00:58:31 on dist-test-slave-j2vl"
server_key: "93e6dd8facc08e2095fc28ae6feb13cd"
server_key_iv: "78e51565bf9b845792476f40e6e8c708"
server_key_version: "encryptionkey@0"
I20260809 00:58:31.774025 12530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:31.813762 12530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:31.818637 12530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:31.818991 12530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:31.840323 12530 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39987
I20260809 00:58:31.840358 12633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39987 every 8 connection(s)
I20260809 00:58:31.842003 12530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:31.846541 12634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:31.848990 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 12530
I20260809 00:58:31.849193 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:31.849815 10116 external_mini_cluster.cc:1428] Setting key b9ccf7a586eaa40abfd6028445c139e7
I20260809 00:58:31.860760 12634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1: Bootstrap starting.
I20260809 00:58:31.864836 12634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:31.865562 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:31.852112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.139:50505 (local address 127.0.0.1:39987)
0809 00:58:31.853090 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:31.853105 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:31.853156 (+ 51us) server_negotiation.cc:407] Connection header received
0809 00:58:31.855290 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:31.855321 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:31.855818 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:31.856279 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:31.856956 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:31.858652 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:31.860180 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:31.861071 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:31.861291 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:31.862815 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:31.862825 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:31.862842 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:31.863480 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:31.863575 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:31.863606 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:31.863802 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:31.864120 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 00:58:31.864319 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":423,"threads_started":1}
I20260809 00:58:31.866199 12634 log.cc:824] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:31.869827 12634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1: No bootstrap required, opened a new log
I20260809 00:58:31.870152 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:31.879837 12634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3422f4ddcf41ab8248cff31b65b6b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } }
I20260809 00:58:31.880568 12634 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:31.880759 12634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3422f4ddcf41ab8248cff31b65b6b1, State: Initialized, Role: FOLLOWER
I20260809 00:58:31.881620 12634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3422f4ddcf41ab8248cff31b65b6b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } }
I20260809 00:58:31.881953 12634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:31.882112 12634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:31.882375 12634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:31.883575 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:31.875059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.139:49207 (local address 127.0.0.1:39987)
0809 00:58:31.875274 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:31.875291 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:31.875753 (+ 462us) server_negotiation.cc:407] Connection header received
0809 00:58:31.876008 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:31.876013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:31.876087 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:31.876214 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:31.876988 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:31.877973 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:31.879531 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:31.880366 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:31.880540 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:31.882392 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:31.882401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:31.882408 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:31.882440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:31.882506 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:31.882537 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:31.882711 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:31.883089 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 00:58:31.883196 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 00:58:31.885643 12634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3422f4ddcf41ab8248cff31b65b6b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } }
I20260809 00:58:31.886437 12634 leader_election.cc:302] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a3422f4ddcf41ab8248cff31b65b6b1; no voters:
I20260809 00:58:31.887200 12634 leader_election.cc:288] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:31.887393 12651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:31.888160 12651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a3422f4ddcf41ab8248cff31b65b6b1, State: Running, Role: LEADER
I20260809 00:58:31.888983 12651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3422f4ddcf41ab8248cff31b65b6b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } }
I20260809 00:58:31.889887 12634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:31.893616 12653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3422f4ddcf41ab8248cff31b65b6b1. Latest consensus state: current_term: 1 leader_uuid: "5a3422f4ddcf41ab8248cff31b65b6b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3422f4ddcf41ab8248cff31b65b6b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } } }
I20260809 00:58:31.892874 12652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a3422f4ddcf41ab8248cff31b65b6b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3422f4ddcf41ab8248cff31b65b6b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39987 } } }
I20260809 00:58:31.894438 12652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:31.900059 12653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:31.903306 12664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:31.908890 12664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:31.920260 12664 catalog_manager.cc:1347] Generated new cluster ID: b4518857ad154fcf9d8325972e97aef5
I20260809 00:58:31.920428 12664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:31.935434 12664 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:31.937546 12664 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:31.949121 12664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a3422f4ddcf41ab8248cff31b65b6b1: Generated new TSK 0
I20260809 00:58:31.950672 12664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:31.970775 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12530
W20260809 00:58:31.972993 12626 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.139:50505
W20260809 00:58:31.973996 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12530: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:58:31.974174 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 12530
W20260809 00:58:32.005915 12507 connection.cc:535] client connection to 127.0.0.1:39987 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39987): Connection reset by peer (error 104)
2026-08-09T00:58:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:32 dist-test-slave-j2vl krb5kdc[12685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:32 dist-test-slave-j2vl krb5kdc[12685](info): set up 2 sockets
Aug 09 00:58:32 dist-test-slave-j2vl krb5kdc[12685](info): commencing operation
krb5kdc: starting...
W20260809 00:58:33.389204 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.362s user 0.005s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:33 dist-test-slave-j2vl krb5kdc[12685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 00:58:33.765228 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:34.166070 12705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:34.166430 12705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:34.166612 12705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:34.174410 12705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:34.174556 12705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:34.174777 12705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:34.174907 12705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:34.175024 12705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:34.185875 12705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36293
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:34.187464 12705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:34.189078 12705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:34.202768 12713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:34.202916 12712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:34.202872 12718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:34.204030 12705 server_base.cc:1034] running on GCE node
I20260809 00:58:34.205003 12705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:34.206732 12705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:34.207964 12705 hybrid_clock.cc:648] HybridClock initialized: now 1786237114207918 us; error 37 us; skew 500 ppm
I20260809 00:58:34.208379 12705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:34 dist-test-slave-j2vl krb5kdc[12685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:34.211853 12705 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 00:58:34.214200 12705 webserver.cc:460] Webserver started at http://127.0.0.1:45175/ using document root <none> and password file <none>
I20260809 00:58:34.215147 12705 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:34.215317 12705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:34.215759 12705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:34.219794 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "6c182265524347a9a3bb77d2fd74d957"
format_stamp: "Formatted at 2026-08-09 00:58:34 on dist-test-slave-j2vl"
server_key: "4c77916b34d5755c02f1e98af1ca6934"
server_key_iv: "8c1239fcc99f472b8555fd15cd259f2a"
server_key_version: "encryptionkey@0"
I20260809 00:58:34.220773 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "6c182265524347a9a3bb77d2fd74d957"
format_stamp: "Formatted at 2026-08-09 00:58:34 on dist-test-slave-j2vl"
server_key: "4c77916b34d5755c02f1e98af1ca6934"
server_key_iv: "8c1239fcc99f472b8555fd15cd259f2a"
server_key_version: "encryptionkey@0"
I20260809 00:58:34.227277 12705 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 00:58:34.231426 12727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:34.232985 12705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 00:58:34.233197 12705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "6c182265524347a9a3bb77d2fd74d957"
format_stamp: "Formatted at 2026-08-09 00:58:34 on dist-test-slave-j2vl"
server_key: "4c77916b34d5755c02f1e98af1ca6934"
server_key_iv: "8c1239fcc99f472b8555fd15cd259f2a"
server_key_version: "encryptionkey@0"
I20260809 00:58:34.233486 12705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:34.240304 12705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:34.245249 12705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:34.245713 12705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:34.267884 12705 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36293
I20260809 00:58:34.267884 12803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36293 every 8 connection(s)
I20260809 00:58:34.269941 12705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:34.273854 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 12705
I20260809 00:58:34.274080 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:34.274503 12804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:34.274700 10116 external_mini_cluster.cc:1428] Setting key 665dbb411eff5f7628dbc3a0dbe0431e
I20260809 00:58:34.283303 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:34.276610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.139:35763 (local address 127.0.0.1:36293)
0809 00:58:34.277450 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:34.277464 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:34.277510 (+ 46us) server_negotiation.cc:407] Connection header received
0809 00:58:34.279249 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:34.279276 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:34.279684 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:34.280128 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:34.281400 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:34.281415 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:34.281430 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:34.281447 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:34.281536 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:34.281559 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:34.281829 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:34.281990 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 00:58:34.282214 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":297,"threads_started":1}
I20260809 00:58:34.287649 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:34.290760 12804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957: Bootstrap starting.
I20260809 00:58:34.295754 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:34.292508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.139:58735 (local address 127.0.0.1:36293)
0809 00:58:34.292791 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:34.292797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:34.293292 (+ 495us) server_negotiation.cc:407] Connection header received
0809 00:58:34.293584 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:34.293589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:34.293648 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:34.293758 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:34.294961 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:34.294966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:34.294969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:34.294986 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:34.295031 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:34.295047 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:34.295145 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:34.295324 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 00:58:34.295392 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 00:58:34.295804 12804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:34.297231 12804 log.cc:824] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:34.300695 12804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957: No bootstrap required, opened a new log
I20260809 00:58:34.309840 12804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c182265524347a9a3bb77d2fd74d957" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36293 } }
I20260809 00:58:34.310411 12804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:34.310547 12804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c182265524347a9a3bb77d2fd74d957, State: Initialized, Role: FOLLOWER
I20260809 00:58:34.311281 12804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c182265524347a9a3bb77d2fd74d957" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36293 } }
I20260809 00:58:34.311630 12804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:34.311825 12804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:34.312040 12804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:34.315282 12804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c182265524347a9a3bb77d2fd74d957" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36293 } }
I20260809 00:58:34.315939 12804 leader_election.cc:302] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c182265524347a9a3bb77d2fd74d957; no voters:
I20260809 00:58:34.316685 12804 leader_election.cc:288] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:34.316903 12817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:34.317752 12817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [term 1 LEADER]: Becoming Leader. State: Replica: 6c182265524347a9a3bb77d2fd74d957, State: Running, Role: LEADER
I20260809 00:58:34.318466 12817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c182265524347a9a3bb77d2fd74d957" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36293 } }
I20260809 00:58:34.319610 12804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:34.323153 12820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c182265524347a9a3bb77d2fd74d957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c182265524347a9a3bb77d2fd74d957" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36293 } } }
I20260809 00:58:34.323153 12821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c182265524347a9a3bb77d2fd74d957. Latest consensus state: current_term: 1 leader_uuid: "6c182265524347a9a3bb77d2fd74d957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c182265524347a9a3bb77d2fd74d957" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36293 } } }
I20260809 00:58:34.323724 12821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:34.323729 12820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:34.331054 12830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:34.336555 12830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:34.346247 12830 catalog_manager.cc:1347] Generated new cluster ID: 88d3f70cfa024fcc8eaf00a4b1fbd09a
I20260809 00:58:34.346403 12830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:34.370606 12830 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:34.371203 12830 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:34.383628 12830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c182265524347a9a3bb77d2fd74d957: Generated new TSK 0
I20260809 00:58:34.384706 12830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:34.403671 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12705
W20260809 00:58:34.404953 12795 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.139:35763
W20260809 00:58:34.405920 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12705: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:58:34.406080 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 12705
2026-08-09T00:58:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:34 dist-test-slave-j2vl krb5kdc[12848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:34 dist-test-slave-j2vl krb5kdc[12848](info): set up 2 sockets
Aug 09 00:58:34 dist-test-slave-j2vl krb5kdc[12848](info): commencing operation
krb5kdc: starting...
W20260809 00:58:35.827759 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.368s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:36 dist-test-slave-j2vl krb5kdc[12848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:58:36.204964 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:42299
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:34985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:42299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:36.604557 12872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:36.604930 12872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:36.605115 12872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:36.613180 12872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:36.613323 12872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:36.613444 12872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:36.613560 12872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:36.613644 12872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:36.624696 12872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:34985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:42299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:42299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:36.626511 12872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:36.628181 12872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:36.641969 12883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:36.642568 12884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:36.642720 12890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:36.642926 12872 server_base.cc:1034] running on GCE node
I20260809 00:58:36.643980 12872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:36.645748 12872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:36.647009 12872 hybrid_clock.cc:648] HybridClock initialized: now 1786237116646944 us; error 63 us; skew 500 ppm
I20260809 00:58:36.647450 12872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:36 dist-test-slave-j2vl krb5kdc[12848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237116, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:36.650929 12872 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:58:36.652603 12872 webserver.cc:460] Webserver started at http://127.9.225.62:46743/ using document root <none> and password file <none>
I20260809 00:58:36.653434 12872 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:36.653566 12872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:36.653860 12872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:36.657284 12872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "3968d7b2aedf48239100243d96552f77"
format_stamp: "Formatted at 2026-08-09 00:58:36 on dist-test-slave-j2vl"
server_key: "8b473bb1f3a504b3ebd060b8352008fb"
server_key_iv: "97ff031196123ede6b9d0821880ec952"
server_key_version: "encryptionkey@0"
I20260809 00:58:36.658223 12872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "3968d7b2aedf48239100243d96552f77"
format_stamp: "Formatted at 2026-08-09 00:58:36 on dist-test-slave-j2vl"
server_key: "8b473bb1f3a504b3ebd060b8352008fb"
server_key_iv: "97ff031196123ede6b9d0821880ec952"
server_key_version: "encryptionkey@0"
I20260809 00:58:36.664338 12872 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260809 00:58:36.668756 12898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:36.670598 12872 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 00:58:36.670804 12872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "3968d7b2aedf48239100243d96552f77"
format_stamp: "Formatted at 2026-08-09 00:58:36 on dist-test-slave-j2vl"
server_key: "8b473bb1f3a504b3ebd060b8352008fb"
server_key_iv: "97ff031196123ede6b9d0821880ec952"
server_key_version: "encryptionkey@0"
I20260809 00:58:36.671010 12872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:36.687992 12872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:36.692930 12872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:36.693228 12872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:36.714215 12872 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:42299
I20260809 00:58:36.714330 12988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:42299 every 8 connection(s)
I20260809 00:58:36.715878 12872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:36.719486 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 12872
I20260809 00:58:36.719687 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:36.720268 10116 external_mini_cluster.cc:1428] Setting key a16d119bd98f2e99c1fa4a921f0a22d1
I20260809 00:58:36.720938 12989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 00:58:36 dist-test-slave-j2vl krb5kdc[12848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:58:36.737857 12989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77: Bootstrap starting.
I20260809 00:58:36.741885 12989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:36.743333 12989 log.cc:824] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:36.747334 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:36.722356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45169 (local address 127.9.225.62:42299)
0809 00:58:36.723245 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:36.723258 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:36.723308 (+ 50us) server_negotiation.cc:407] Connection header received
0809 00:58:36.725009 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:36.725035 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:36.725595 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:36.725984 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:36.728133 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:36.730007 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:36.731878 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:36.732631 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:36.732798 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:36.738167 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:36.738176 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:36.738194 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:36.738245 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:36.741677 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:36.742759 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:36.742769 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:36.742777 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:36.743024 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:36.743586 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:36.743592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:36.743595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:36.744378 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:36.744650 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:36.745475 (+ 825us) server_negotiation.cc:299] Negotiation successful
0809 00:58:36.745871 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":410,"threads_started":1}
I20260809 00:58:36.747551 12989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77: No bootstrap required, opened a new log
I20260809 00:58:36.751992 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:36.756812 12989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3968d7b2aedf48239100243d96552f77" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42299 } }
I20260809 00:58:36.757465 12989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:36.757589 12989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3968d7b2aedf48239100243d96552f77, State: Initialized, Role: FOLLOWER
I20260809 00:58:36.758359 12989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3968d7b2aedf48239100243d96552f77" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42299 } }
I20260809 00:58:36.758616 12989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:36.758791 12989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:36.758986 12989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:36.761593 12989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3968d7b2aedf48239100243d96552f77" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42299 } }
I20260809 00:58:36.762213 12989 leader_election.cc:302] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3968d7b2aedf48239100243d96552f77; no voters:
I20260809 00:58:36.762979 12989 leader_election.cc:288] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:36.763129 13005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:36.764101 13005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [term 1 LEADER]: Becoming Leader. State: Replica: 3968d7b2aedf48239100243d96552f77, State: Running, Role: LEADER
I20260809 00:58:36.764782 13005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3968d7b2aedf48239100243d96552f77" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42299 } }
I20260809 00:58:36.765813 12989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:36.768642 13006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3968d7b2aedf48239100243d96552f77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3968d7b2aedf48239100243d96552f77" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42299 } } }
I20260809 00:58:36.769083 13006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:36.770239 13008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3968d7b2aedf48239100243d96552f77. Latest consensus state: current_term: 1 leader_uuid: "3968d7b2aedf48239100243d96552f77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3968d7b2aedf48239100243d96552f77" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42299 } } }
I20260809 00:58:36.770608 13008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:36.773298 13019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:36.783288 13019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:36.783886 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:36.756997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51231 (local address 127.9.225.62:42299)
0809 00:58:36.757224 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:36.757232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:36.757786 (+ 554us) server_negotiation.cc:407] Connection header received
0809 00:58:36.758013 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:36.758023 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:36.758096 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:36.758206 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:36.760000 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:36.760844 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:36.762370 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:36.762877 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:36.763065 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:36.771612 (+ 8547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:36.771623 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:36.771630 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:36.771692 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:36.777174 (+ 5482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:36.778771 (+ 1597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:36.778780 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:36.778787 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:36.779595 (+ 808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:36.780342 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:36.780347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:36.780351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:36.780750 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:36.782655 (+ 1905us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:36.782946 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 00:58:36.783269 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 00:58:36.800530 13019 catalog_manager.cc:1347] Generated new cluster ID: 1a435df2c1484ded92d08a7e28b125bf
I20260809 00:58:36.800740 13019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:36.821635 13019 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:36.823282 13019 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:36.833855 13019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3968d7b2aedf48239100243d96552f77: Generated new TSK 0
I20260809 00:58:36.835037 13019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:36.851089 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 12872
I20260809 00:58:36.852398 12981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:36.952937 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 12872
W20260809 00:58:36.985988 12844 connection.cc:535] client connection to 127.9.225.62:42299 recv error: Network error: failed to read from TLS socket (remote: 127.9.225.62:42299): Connection reset by peer (error 104)
2026-08-09T00:58:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2545 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:37 dist-test-slave-j2vl krb5kdc[13043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:37 dist-test-slave-j2vl krb5kdc[13043](info): set up 2 sockets
Aug 09 00:58:37 dist-test-slave-j2vl krb5kdc[13043](info): commencing operation
krb5kdc: starting...
W20260809 00:58:38.353060 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.347s user 0.003s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:38 dist-test-slave-j2vl krb5kdc[13043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:58:38.733945 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43285
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:46219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:43285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:39.126443 13074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:39.126874 13074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:39.127063 13074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:39.134799 13074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:39.135006 13074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:39.135123 13074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:39.135239 13074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:39.135339 13074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:39.146104 13074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:46219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:43285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:43285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:39.147749 13074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:39.149322 13074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:39.163134 13089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:39.163288 13087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:39.163316 13084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:39.164556 13074 server_base.cc:1034] running on GCE node
I20260809 00:58:39.165647 13074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:39.167451 13074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:39.168715 13074 hybrid_clock.cc:648] HybridClock initialized: now 1786237119168671 us; error 55 us; skew 500 ppm
I20260809 00:58:39.169129 13074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:39 dist-test-slave-j2vl krb5kdc[13043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237119, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:39.173233 13074 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:58:39.175697 13074 webserver.cc:460] Webserver started at http://127.9.225.62:38335/ using document root <none> and password file <none>
I20260809 00:58:39.176512 13074 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:39.176637 13074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:39.176934 13074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:39.180371 13074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "1e9537ad3e2c47e0963552d7af4b6c12"
format_stamp: "Formatted at 2026-08-09 00:58:39 on dist-test-slave-j2vl"
server_key: "9c92409687607dda45061578ee615cc8"
server_key_iv: "8b9a1e53d3c021535d3d54baeac47ae8"
server_key_version: "encryptionkey@0"
I20260809 00:58:39.181216 13074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "1e9537ad3e2c47e0963552d7af4b6c12"
format_stamp: "Formatted at 2026-08-09 00:58:39 on dist-test-slave-j2vl"
server_key: "9c92409687607dda45061578ee615cc8"
server_key_iv: "8b9a1e53d3c021535d3d54baeac47ae8"
server_key_version: "encryptionkey@0"
I20260809 00:58:39.187561 13074 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:58:39.191802 13100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:39.193574 13074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 00:58:39.193828 13074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "1e9537ad3e2c47e0963552d7af4b6c12"
format_stamp: "Formatted at 2026-08-09 00:58:39 on dist-test-slave-j2vl"
server_key: "9c92409687607dda45061578ee615cc8"
server_key_iv: "8b9a1e53d3c021535d3d54baeac47ae8"
server_key_version: "encryptionkey@0"
I20260809 00:58:39.194048 13074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:39.228940 13074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:39.234144 13074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:39.234481 13074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:39.255939 13074 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:43285
I20260809 00:58:39.256004 13190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:43285 every 8 connection(s)
I20260809 00:58:39.257750 13074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:39.262422 13192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:39.264130 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 13074
I20260809 00:58:39.264346 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:39.264957 10116 external_mini_cluster.cc:1428] Setting key b6b86abcad4a57f06f2c3f52c44b76e2
I20260809 00:58:39.280292 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12: Bootstrap starting.
I20260809 00:58:39.282521 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:39.267187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60599 (local address 127.9.225.62:43285)
0809 00:58:39.268146 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:39.268159 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:39.268208 (+ 49us) server_negotiation.cc:407] Connection header received
0809 00:58:39.270091 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:39.270129 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:39.270668 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:39.271090 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:39.271792 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:39.273171 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:39.275496 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:39.276560 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:39.276771 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:39.278799 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:39.278813 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:39.278840 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:39.279699 (+ 859us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:39.279866 (+ 167us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:39.279907 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:39.280088 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:39.280584 (+ 496us) server_negotiation.cc:299] Negotiation successful
0809 00:58:39.280885 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":437,"threads_started":1}
I20260809 00:58:39.286331 13192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:39.288342 13192 log.cc:824] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:39.289227 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:39.294077 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12: No bootstrap required, opened a new log
I20260809 00:58:39.306353 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:39.296555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51291 (local address 127.9.225.62:43285)
0809 00:58:39.296846 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:39.296853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:39.298150 (+ 1297us) server_negotiation.cc:407] Connection header received
0809 00:58:39.298415 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:39.298420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:39.298525 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:39.298625 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:39.299576 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:39.300595 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:39.302942 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:39.303572 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:39.303704 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:39.305302 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:39.305309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:39.305313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:39.305393 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:39.305475 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:39.305498 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:39.305613 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:39.305876 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 00:58:39.305967 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 00:58:39.306900 13192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e9537ad3e2c47e0963552d7af4b6c12" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43285 } }
I20260809 00:58:39.307369 13192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:39.307495 13192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e9537ad3e2c47e0963552d7af4b6c12, State: Initialized, Role: FOLLOWER
I20260809 00:58:39.308192 13192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e9537ad3e2c47e0963552d7af4b6c12" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43285 } }
I20260809 00:58:39.308495 13192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:39.308652 13192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:39.308852 13192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:39.311380 13192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e9537ad3e2c47e0963552d7af4b6c12" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43285 } }
I20260809 00:58:39.311981 13192 leader_election.cc:302] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e9537ad3e2c47e0963552d7af4b6c12; no voters:
I20260809 00:58:39.312661 13192 leader_election.cc:288] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:39.312827 13204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:39.313621 13204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [term 1 LEADER]: Becoming Leader. State: Replica: 1e9537ad3e2c47e0963552d7af4b6c12, State: Running, Role: LEADER
I20260809 00:58:39.314296 13204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e9537ad3e2c47e0963552d7af4b6c12" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43285 } }
I20260809 00:58:39.315115 13192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:39.317914 13205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e9537ad3e2c47e0963552d7af4b6c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e9537ad3e2c47e0963552d7af4b6c12" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43285 } } }
I20260809 00:58:39.318363 13205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:39.317873 13206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e9537ad3e2c47e0963552d7af4b6c12. Latest consensus state: current_term: 1 leader_uuid: "1e9537ad3e2c47e0963552d7af4b6c12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e9537ad3e2c47e0963552d7af4b6c12" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 43285 } } }
I20260809 00:58:39.319173 13206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:39.326200 13215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:39.332500 13215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:39.342969 13215 catalog_manager.cc:1347] Generated new cluster ID: e0c915228d13466685d8544ac9d47905
I20260809 00:58:39.343145 13215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:39.376549 13215 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:39.378050 13215 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:39.389770 13215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e9537ad3e2c47e0963552d7af4b6c12: Generated new TSK 0
I20260809 00:58:39.390799 13215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:39.404274 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13074
W20260809 00:58:39.406978 13183 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60599
W20260809 00:58:39.408528 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13074: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:58:39.408838 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 13074
W20260809 00:58:39.474073 13041 connection.cc:535] client connection to 127.9.225.62:43285 recv error: Network error: failed to read from TLS socket (remote: 127.9.225.62:43285): Connection reset by peer (error 104)
2026-08-09T00:58:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2487 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:39 dist-test-slave-j2vl krb5kdc[13235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:39 dist-test-slave-j2vl krb5kdc[13235](info): set up 2 sockets
Aug 09 00:58:39 dist-test-slave-j2vl krb5kdc[13235](info): commencing operation
krb5kdc: starting...
W20260809 00:58:40.861176 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.367s user 0.007s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:41 dist-test-slave-j2vl krb5kdc[13235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:58:41.243412 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:42209
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:41509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:42209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:41.637775 13255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:41.638123 13255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:41.638233 13255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:41.646114 13255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:41.646277 13255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:41.646329 13255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:41.646386 13255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:41.646426 13255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:41.657416 13255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:41509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:42209
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:42209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:41.659065 13255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:41.660611 13255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:41.674654 13263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:41.674764 13262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:41.674773 13266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:41.675812 13255 server_base.cc:1034] running on GCE node
I20260809 00:58:41.676904 13255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:41.678830 13255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:41.680137 13255 hybrid_clock.cc:648] HybridClock initialized: now 1786237121680067 us; error 69 us; skew 500 ppm
I20260809 00:58:41.680747 13255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:41 dist-test-slave-j2vl krb5kdc[13235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237121, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:41.684383 13255 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:58:41.686342 13255 webserver.cc:460] Webserver started at http://127.9.225.62:33583/ using document root <none> and password file <none>
I20260809 00:58:41.687178 13255 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:41.687345 13255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:41.687700 13255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:41.691406 13255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "67117b34ae7c4cacb0bfc26ffb405d6f"
format_stamp: "Formatted at 2026-08-09 00:58:41 on dist-test-slave-j2vl"
server_key: "6c5581f78ed0515b002e770a69bdf6c1"
server_key_iv: "0237455456820de41e71b13fa94be93b"
server_key_version: "encryptionkey@0"
I20260809 00:58:41.692358 13255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "67117b34ae7c4cacb0bfc26ffb405d6f"
format_stamp: "Formatted at 2026-08-09 00:58:41 on dist-test-slave-j2vl"
server_key: "6c5581f78ed0515b002e770a69bdf6c1"
server_key_iv: "0237455456820de41e71b13fa94be93b"
server_key_version: "encryptionkey@0"
I20260809 00:58:41.699385 13255 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260809 00:58:41.704185 13282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:41.706076 13255 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 00:58:41.706334 13255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "67117b34ae7c4cacb0bfc26ffb405d6f"
format_stamp: "Formatted at 2026-08-09 00:58:41 on dist-test-slave-j2vl"
server_key: "6c5581f78ed0515b002e770a69bdf6c1"
server_key_iv: "0237455456820de41e71b13fa94be93b"
server_key_version: "encryptionkey@0"
I20260809 00:58:41.706620 13255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:41.713660 13255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:41.719249 13255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:41.719630 13255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:41.740859 13255 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:42209
I20260809 00:58:41.740891 13368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:42209 every 8 connection(s)
I20260809 00:58:41.742604 13255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:41.747087 13369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:41.752436 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 13255
I20260809 00:58:41.752671 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:41.753134 10116 external_mini_cluster.cc:1428] Setting key 467fabdda4fa7b712a045d204397dceb
I20260809 00:58:41.760454 13369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f: Bootstrap starting.
I20260809 00:58:41.762645 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:41.755327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60475 (local address 127.9.225.62:42209)
0809 00:58:41.756320 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:41.756337 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:41.756385 (+ 48us) server_negotiation.cc:407] Connection header received
0809 00:58:41.758521 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:41.758554 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:41.758965 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:41.759412 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:41.760688 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:41.760710 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:41.760736 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:41.760770 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:41.760855 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:41.760883 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:41.761227 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:41.761422 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 00:58:41.761630 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":470,"threads_started":1}
I20260809 00:58:41.764554 13369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:41.765982 13369 log.cc:824] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:41.767105 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:41.769285 13369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f: No bootstrap required, opened a new log
I20260809 00:58:41.779206 13369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67117b34ae7c4cacb0bfc26ffb405d6f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42209 } }
I20260809 00:58:41.779771 13369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:41.779826 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:41.772141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38513 (local address 127.9.225.62:42209)
0809 00:58:41.772386 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:41.772400 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:41.772792 (+ 392us) server_negotiation.cc:407] Connection header received
0809 00:58:41.773036 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:41.773040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:41.773103 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:41.773228 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:41.779010 (+ 5782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:41.779016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:41.779019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:41.779043 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:41.779102 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:41.779121 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:41.779282 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:41.779408 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 00:58:41.779485 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 00:58:41.779904 13369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67117b34ae7c4cacb0bfc26ffb405d6f, State: Initialized, Role: FOLLOWER
I20260809 00:58:41.780705 13369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67117b34ae7c4cacb0bfc26ffb405d6f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42209 } }
I20260809 00:58:41.781019 13369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:41.781206 13369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:41.781481 13369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:41.784348 13369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67117b34ae7c4cacb0bfc26ffb405d6f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42209 } }
I20260809 00:58:41.784912 13369 leader_election.cc:302] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67117b34ae7c4cacb0bfc26ffb405d6f; no voters:
I20260809 00:58:41.785554 13369 leader_election.cc:288] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:41.785706 13380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:41.786463 13380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [term 1 LEADER]: Becoming Leader. State: Replica: 67117b34ae7c4cacb0bfc26ffb405d6f, State: Running, Role: LEADER
I20260809 00:58:41.787071 13380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67117b34ae7c4cacb0bfc26ffb405d6f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42209 } }
I20260809 00:58:41.787990 13369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:41.790059 13382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67117b34ae7c4cacb0bfc26ffb405d6f. Latest consensus state: current_term: 1 leader_uuid: "67117b34ae7c4cacb0bfc26ffb405d6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67117b34ae7c4cacb0bfc26ffb405d6f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42209 } } }
I20260809 00:58:41.790597 13382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:41.791771 13389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:41.794656 13381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67117b34ae7c4cacb0bfc26ffb405d6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67117b34ae7c4cacb0bfc26ffb405d6f" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42209 } } }
I20260809 00:58:41.795073 13381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:41.800052 13389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:41.813431 13389 catalog_manager.cc:1347] Generated new cluster ID: 07ba721b4c094a70a972817566669c13
I20260809 00:58:41.813599 13389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:41.832818 13389 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:41.833601 13389 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:41.847259 13389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67117b34ae7c4cacb0bfc26ffb405d6f: Generated new TSK 0
I20260809 00:58:41.848323 13389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:41.856248 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13255
W20260809 00:58:41.857610 13363 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60475
W20260809 00:58:41.858559 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13255: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:58:41.858765 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 13255
2026-08-09T00:58:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:41 dist-test-slave-j2vl krb5kdc[13412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:41 dist-test-slave-j2vl krb5kdc[13412](info): set up 2 sockets
Aug 09 00:58:41 dist-test-slave-j2vl krb5kdc[13412](info): commencing operation
krb5kdc: starting...
W20260809 00:58:43.258358 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.349s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:43 dist-test-slave-j2vl krb5kdc[13412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:58:43.649240 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39171
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:39171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:44.061868 13430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:44.062256 13430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:44.062361 13430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:44.070732 13430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:44.070897 13430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:44.070951 13430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:44.071019 13430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:44.071067 13430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:44.082093 13430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:39171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:39171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:44.083595 13430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:44.084992 13430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:44.098726 13442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:44.098759 13446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:44.098964 13444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:44.103549 13430 server_base.cc:1034] running on GCE node
I20260809 00:58:44.104626 13430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:44.106441 13430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:44.107703 13430 hybrid_clock.cc:648] HybridClock initialized: now 1786237124107653 us; error 47 us; skew 500 ppm
I20260809 00:58:44.108109 13430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:44 dist-test-slave-j2vl krb5kdc[13412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237124, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:44.111655 13430 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:58:44.113575 13430 webserver.cc:460] Webserver started at http://127.9.225.62:37309/ using document root <none> and password file <none>
I20260809 00:58:44.114430 13430 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:44.114585 13430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:44.114946 13430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:44.118467 13430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "e75d79f64d1244188d048daa2b72e77b"
format_stamp: "Formatted at 2026-08-09 00:58:44 on dist-test-slave-j2vl"
server_key: "17d4c1189c545915e9fee9308689f912"
server_key_iv: "295f7986451c1fd8f9918865d2957bcb"
server_key_version: "encryptionkey@0"
I20260809 00:58:44.119380 13430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "e75d79f64d1244188d048daa2b72e77b"
format_stamp: "Formatted at 2026-08-09 00:58:44 on dist-test-slave-j2vl"
server_key: "17d4c1189c545915e9fee9308689f912"
server_key_iv: "295f7986451c1fd8f9918865d2957bcb"
server_key_version: "encryptionkey@0"
I20260809 00:58:44.125672 13430 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 00:58:44.130483 13459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:44.132476 13430 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:58:44.132733 13430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "e75d79f64d1244188d048daa2b72e77b"
format_stamp: "Formatted at 2026-08-09 00:58:44 on dist-test-slave-j2vl"
server_key: "17d4c1189c545915e9fee9308689f912"
server_key_iv: "295f7986451c1fd8f9918865d2957bcb"
server_key_version: "encryptionkey@0"
I20260809 00:58:44.132980 13430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:44.147650 13430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:44.152956 13430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:44.153326 13430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:44.175745 13430 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:39171
I20260809 00:58:44.175765 13536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:39171 every 8 connection(s)
I20260809 00:58:44.177543 13430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:44.180915 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 13430
I20260809 00:58:44.181115 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:44.181773 10116 external_mini_cluster.cc:1428] Setting key 3dfeeb32b67e733fc3d4c31aaca3d338
I20260809 00:58:44.182467 13537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 00:58:44 dist-test-slave-j2vl krb5kdc[13412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:58:44.197459 13537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b: Bootstrap starting.
I20260809 00:58:44.201592 13537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:44.202811 13537 log.cc:824] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:44.206250 13537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b: No bootstrap required, opened a new log
I20260809 00:58:44.207090 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:44.183936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59577 (local address 127.9.225.62:39171)
0809 00:58:44.184808 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:44.184824 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:44.184870 (+ 46us) server_negotiation.cc:407] Connection header received
0809 00:58:44.186643 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:44.186673 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:44.187180 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:44.187677 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:44.189429 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:44.190925 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:44.192864 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:44.193766 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:44.194009 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:44.198784 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:44.198795 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:44.198814 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:44.198865 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:44.202359 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:44.203236 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:44.203245 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:44.203252 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:44.203475 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:44.203903 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:44.203914 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:44.203916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:44.204532 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:44.204851 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:44.205423 (+ 572us) server_negotiation.cc:299] Negotiation successful
0809 00:58:44.205883 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":378,"threads_started":1}
I20260809 00:58:44.211604 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:44.215709 13537 raft_consensus.cc:348] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d79f64d1244188d048daa2b72e77b" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39171 } }
I20260809 00:58:44.216193 13537 raft_consensus.cc:374] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:44.216359 13537 raft_consensus.cc:729] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75d79f64d1244188d048daa2b72e77b, State: Initialized, Role: FOLLOWER
I20260809 00:58:44.217088 13537 consensus_queue.cc:260] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d79f64d1244188d048daa2b72e77b" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39171 } }
I20260809 00:58:44.217487 13537 raft_consensus.cc:388] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:44.217675 13537 raft_consensus.cc:482] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:44.217883 13537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:44.221140 13537 raft_consensus.cc:504] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d79f64d1244188d048daa2b72e77b" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39171 } }
I20260809 00:58:44.221864 13537 leader_election.cc:302] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e75d79f64d1244188d048daa2b72e77b; no voters:
I20260809 00:58:44.222651 13537 leader_election.cc:288] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:44.222846 13550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:44.224205 13550 raft_consensus.cc:686] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [term 1 LEADER]: Becoming Leader. State: Replica: e75d79f64d1244188d048daa2b72e77b, State: Running, Role: LEADER
I20260809 00:58:44.224984 13550 consensus_queue.cc:237] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d79f64d1244188d048daa2b72e77b" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39171 } }
I20260809 00:58:44.225293 13537 sys_catalog.cc:564] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:44.229010 13552 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e75d79f64d1244188d048daa2b72e77b. Latest consensus state: current_term: 1 leader_uuid: "e75d79f64d1244188d048daa2b72e77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d79f64d1244188d048daa2b72e77b" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39171 } } }
I20260809 00:58:44.229584 13552 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:44.230444 13551 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e75d79f64d1244188d048daa2b72e77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75d79f64d1244188d048daa2b72e77b" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 39171 } } }
I20260809 00:58:44.230897 13551 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:44.238445 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:44.216411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60627 (local address 127.9.225.62:39171)
0809 00:58:44.216583 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:44.216588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:44.217402 (+ 814us) server_negotiation.cc:407] Connection header received
0809 00:58:44.217619 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:44.217626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:44.217711 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:44.217836 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:44.219875 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:44.220598 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:44.222470 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:44.223453 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:44.223638 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:44.227541 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:44.227551 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:44.227560 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:44.227617 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:44.232315 (+ 4698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:44.233772 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:44.233781 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:44.233787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:44.234051 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:44.234703 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:44.234711 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:44.234731 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:44.235306 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:44.236982 (+ 1676us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:44.237221 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 00:58:44.237860 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 00:58:44.241636 13563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:44.248771 13563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:44.260408 13563 catalog_manager.cc:1347] Generated new cluster ID: e8582d2d71c542cd90b203f9dc3fadb0
I20260809 00:58:44.260564 13563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:44.273931 13563 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:44.275525 13563 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:44.285501 13563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e75d79f64d1244188d048daa2b72e77b: Generated new TSK 0
I20260809 00:58:44.286497 13563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:44.297398 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13430
I20260809 00:58:44.299003 13531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:44.396296 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 13430
2026-08-09T00:58:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2534 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:44 dist-test-slave-j2vl krb5kdc[13578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:44 dist-test-slave-j2vl krb5kdc[13578](info): set up 2 sockets
Aug 09 00:58:44 dist-test-slave-j2vl krb5kdc[13578](info): commencing operation
krb5kdc: starting...
W20260809 00:58:45.791817 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.348s user 0.002s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:46 dist-test-slave-j2vl krb5kdc[13578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:58:46.179028 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:42951
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:42951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:46.592388 13597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:46.592834 13597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:46.593051 13597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:46.600865 13597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:46.601079 13597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:46.601202 13597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:46.601303 13597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:46.601421 13597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:46.612169 13597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:42951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:42951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:46.613816 13597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:46.615365 13597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:46.628973 13604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:46.628996 13606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:46.629048 13610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:46.629980 13597 server_base.cc:1034] running on GCE node
I20260809 00:58:46.630951 13597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:46.632864 13597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:46.634158 13597 hybrid_clock.cc:648] HybridClock initialized: now 1786237126634112 us; error 66 us; skew 500 ppm
I20260809 00:58:46.634617 13597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:46 dist-test-slave-j2vl krb5kdc[13578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237126, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:46.638268 13597 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:58:46.640055 13597 webserver.cc:460] Webserver started at http://127.9.225.62:46437/ using document root <none> and password file <none>
I20260809 00:58:46.640825 13597 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:46.640938 13597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:46.641225 13597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:46.644975 13597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "aa4e6ea0163a4c38b8adc634e680db81"
format_stamp: "Formatted at 2026-08-09 00:58:46 on dist-test-slave-j2vl"
server_key: "26ecbecaeb48f72b05a136b3856c779a"
server_key_iv: "f8be7fdfca222e1d18c988d0831cce31"
server_key_version: "encryptionkey@0"
I20260809 00:58:46.646003 13597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "aa4e6ea0163a4c38b8adc634e680db81"
format_stamp: "Formatted at 2026-08-09 00:58:46 on dist-test-slave-j2vl"
server_key: "26ecbecaeb48f72b05a136b3856c779a"
server_key_iv: "f8be7fdfca222e1d18c988d0831cce31"
server_key_version: "encryptionkey@0"
I20260809 00:58:46.652508 13597 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 00:58:46.656807 13620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:46.658548 13597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 00:58:46.658790 13597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "aa4e6ea0163a4c38b8adc634e680db81"
format_stamp: "Formatted at 2026-08-09 00:58:46 on dist-test-slave-j2vl"
server_key: "26ecbecaeb48f72b05a136b3856c779a"
server_key_iv: "f8be7fdfca222e1d18c988d0831cce31"
server_key_version: "encryptionkey@0"
I20260809 00:58:46.659030 13597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:46.675983 13597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:46.682212 13597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:46.682581 13597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:46.704177 13597 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:42951
I20260809 00:58:46.704210 13703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:42951 every 8 connection(s)
I20260809 00:58:46.705876 13597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:46.709043 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 13597
I20260809 00:58:46.709224 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:46.709721 10116 external_mini_cluster.cc:1428] Setting key 0cc694e0c162dd012f8b1c99af465db0
I20260809 00:58:46.710505 13704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:46.726569 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:46.711819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32975 (local address 127.9.225.62:42951)
0809 00:58:46.712730 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:46.712748 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:46.712802 (+ 54us) server_negotiation.cc:407] Connection header received
0809 00:58:46.714666 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:46.714699 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:46.715367 (+ 668us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:46.715906 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:46.716506 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:46.718693 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:46.720166 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:46.721497 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:46.721730 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:46.723101 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:46.723113 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:46.723137 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:46.724029 (+ 892us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:46.724140 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:46.724180 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:46.724323 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:46.724603 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 00:58:46.724978 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":406,"threads_started":1}
I20260809 00:58:46.727936 13704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81: Bootstrap starting.
I20260809 00:58:46.731456 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:46.732326 13704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:46.733594 13704 log.cc:824] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:46.737874 13704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81: No bootstrap required, opened a new log
I20260809 00:58:46.744118 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:46.736732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52215 (local address 127.9.225.62:42951)
0809 00:58:46.736947 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:46.736953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:46.737216 (+ 263us) server_negotiation.cc:407] Connection header received
0809 00:58:46.737425 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:46.737431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:46.737516 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:46.737641 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:46.738354 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:46.739515 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:46.740899 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:46.741721 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:46.741867 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:46.743166 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:46.743173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:46.743177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:46.743203 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:46.743280 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:46.743310 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:46.743422 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:46.743589 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 00:58:46.743720 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 00:58:46.748284 13704 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4e6ea0163a4c38b8adc634e680db81" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42951 } }
I20260809 00:58:46.748741 13704 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:46.748833 13704 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa4e6ea0163a4c38b8adc634e680db81, State: Initialized, Role: FOLLOWER
I20260809 00:58:46.749524 13704 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4e6ea0163a4c38b8adc634e680db81" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42951 } }
I20260809 00:58:46.749784 13704 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:46.749900 13704 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:46.750046 13704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:46.752866 13704 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4e6ea0163a4c38b8adc634e680db81" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42951 } }
I20260809 00:58:46.753621 13704 leader_election.cc:302] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa4e6ea0163a4c38b8adc634e680db81; no voters:
I20260809 00:58:46.754328 13704 leader_election.cc:288] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:46.754622 13720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:46.755406 13720 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [term 1 LEADER]: Becoming Leader. State: Replica: aa4e6ea0163a4c38b8adc634e680db81, State: Running, Role: LEADER
I20260809 00:58:46.756220 13720 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4e6ea0163a4c38b8adc634e680db81" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42951 } }
I20260809 00:58:46.758004 13704 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:46.759773 13721 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa4e6ea0163a4c38b8adc634e680db81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4e6ea0163a4c38b8adc634e680db81" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42951 } } }
I20260809 00:58:46.759750 13723 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa4e6ea0163a4c38b8adc634e680db81. Latest consensus state: current_term: 1 leader_uuid: "aa4e6ea0163a4c38b8adc634e680db81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa4e6ea0163a4c38b8adc634e680db81" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 42951 } } }
I20260809 00:58:46.760340 13721 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:46.760340 13723 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:46.761466 13726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:46.772269 13726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:46.783346 13726 catalog_manager.cc:1347] Generated new cluster ID: ad0971958dc047ceb62109f7d202ccab
I20260809 00:58:46.783529 13726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:46.807662 13726 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:46.809860 13726 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:46.819347 13726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa4e6ea0163a4c38b8adc634e680db81: Generated new TSK 0
I20260809 00:58:46.820565 13726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:46.836479 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13597
W20260809 00:58:46.837898 13698 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32975
W20260809 00:58:46.838831 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13597: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:58:46.839020 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 13597
2026-08-09T00:58:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:46 dist-test-slave-j2vl krb5kdc[13748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:46 dist-test-slave-j2vl krb5kdc[13748](info): set up 2 sockets
Aug 09 00:58:46 dist-test-slave-j2vl krb5kdc[13748](info): commencing operation
krb5kdc: starting...
W20260809 00:58:48.252760 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.362s user 0.002s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:48 dist-test-slave-j2vl krb5kdc[13748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:58:48.639063 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:36745
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:36745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:49.027887 13773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:49.028241 13773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:49.028337 13773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:49.036078 13773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:49.036221 13773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:49.036270 13773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:49.036335 13773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:49.036382 13773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:49.048436 13773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:36745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:36745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:49.050143 13773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:49.051745 13773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:49.065672 13785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:49.065702 13782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:49.065728 13783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:49.071516 13773 server_base.cc:1034] running on GCE node
I20260809 00:58:49.072595 13773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:49.074425 13773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:49.075706 13773 hybrid_clock.cc:648] HybridClock initialized: now 1786237129075650 us; error 58 us; skew 500 ppm
I20260809 00:58:49.076143 13773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:49 dist-test-slave-j2vl krb5kdc[13748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237129, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:49.079835 13773 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:58:49.081666 13773 webserver.cc:460] Webserver started at http://127.9.225.62:44659/ using document root <none> and password file <none>
I20260809 00:58:49.082490 13773 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:49.082607 13773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:49.082911 13773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:49.086362 13773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "3ab4a5f223244a0bbebb1d15f80a2ea5"
format_stamp: "Formatted at 2026-08-09 00:58:49 on dist-test-slave-j2vl"
server_key: "848b1665ae6b3811260bd5cf361073ac"
server_key_iv: "f3105f5decf63d550f1dfe17050ee674"
server_key_version: "encryptionkey@0"
I20260809 00:58:49.087200 13773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "3ab4a5f223244a0bbebb1d15f80a2ea5"
format_stamp: "Formatted at 2026-08-09 00:58:49 on dist-test-slave-j2vl"
server_key: "848b1665ae6b3811260bd5cf361073ac"
server_key_iv: "f3105f5decf63d550f1dfe17050ee674"
server_key_version: "encryptionkey@0"
I20260809 00:58:49.093580 13773 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260809 00:58:49.097896 13795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:49.099598 13773 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 00:58:49.099789 13773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "3ab4a5f223244a0bbebb1d15f80a2ea5"
format_stamp: "Formatted at 2026-08-09 00:58:49 on dist-test-slave-j2vl"
server_key: "848b1665ae6b3811260bd5cf361073ac"
server_key_iv: "f3105f5decf63d550f1dfe17050ee674"
server_key_version: "encryptionkey@0"
I20260809 00:58:49.100018 13773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:49.106676 13773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:49.111733 13773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:49.112059 13773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:49.132879 13773 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:36745
I20260809 00:58:49.132931 13870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:36745 every 8 connection(s)
I20260809 00:58:49.134824 13773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:49.137570 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 13773
I20260809 00:58:49.137833 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:49.138388 10116 external_mini_cluster.cc:1428] Setting key aea13c4f8441123b0c21ffe51c3a5986
I20260809 00:58:49.140039 13874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:49.146993 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:49.140417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48469 (local address 127.9.225.62:36745)
0809 00:58:49.141297 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:49.141310 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:49.141407 (+ 97us) server_negotiation.cc:407] Connection header received
0809 00:58:49.143108 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:49.143134 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:49.143531 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:49.143912 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:49.145122 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:49.145129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:49.145144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:49.145160 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:49.145238 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:49.145262 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:49.145523 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:49.145634 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 00:58:49.145901 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":398,"threads_started":1}
I20260809 00:58:49.151826 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:49.155405 13874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5: Bootstrap starting.
I20260809 00:58:49.160450 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:49.157699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54515 (local address 127.9.225.62:36745)
0809 00:58:49.157945 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:49.157953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:49.158254 (+ 301us) server_negotiation.cc:407] Connection header received
0809 00:58:49.158460 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:49.158465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:49.158525 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:49.158619 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:49.159703 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:49.159708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:49.159711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 00:58:49.159728 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:49.159772 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 00:58:49.159788 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:49.159885 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:49.160034 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 00:58:49.160088 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260809 00:58:49.160490 13874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:49.162129 13874 log.cc:824] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:49.165434 13874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5: No bootstrap required, opened a new log
I20260809 00:58:49.174273 13874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab4a5f223244a0bbebb1d15f80a2ea5" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36745 } }
I20260809 00:58:49.174737 13874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 00:58:49.174813 13874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab4a5f223244a0bbebb1d15f80a2ea5, State: Initialized, Role: FOLLOWER
I20260809 00:58:49.175392 13874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab4a5f223244a0bbebb1d15f80a2ea5" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36745 } }
I20260809 00:58:49.175608 13874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 00:58:49.175726 13874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 00:58:49.175869 13874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:49.178368 13874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab4a5f223244a0bbebb1d15f80a2ea5" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36745 } }
I20260809 00:58:49.178942 13874 leader_election.cc:302] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ab4a5f223244a0bbebb1d15f80a2ea5; no voters:
I20260809 00:58:49.179586 13874 leader_election.cc:288] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 00:58:49.179766 13886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:49.181173 13886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [term 1 LEADER]: Becoming Leader. State: Replica: 3ab4a5f223244a0bbebb1d15f80a2ea5, State: Running, Role: LEADER
I20260809 00:58:49.181927 13886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab4a5f223244a0bbebb1d15f80a2ea5" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36745 } }
I20260809 00:58:49.182268 13874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:49.185468 13887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ab4a5f223244a0bbebb1d15f80a2ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab4a5f223244a0bbebb1d15f80a2ea5" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36745 } } }
I20260809 00:58:49.186089 13887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:49.185477 13888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ab4a5f223244a0bbebb1d15f80a2ea5. Latest consensus state: current_term: 1 leader_uuid: "3ab4a5f223244a0bbebb1d15f80a2ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab4a5f223244a0bbebb1d15f80a2ea5" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36745 } } }
I20260809 00:58:49.186538 13888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5 [sys.catalog]: This master's current role is: LEADER
W20260809 00:58:49.195544 13906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 00:58:49.195693 13906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 00:58:49.195850 13899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:49.198390 13899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:49.209267 13899 catalog_manager.cc:1347] Generated new cluster ID: 28a02f389ea34f9c9292dcee97398b70
I20260809 00:58:49.209489 13899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:49.229085 13899 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:49.229640 13899 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:49.236307 13899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ab4a5f223244a0bbebb1d15f80a2ea5: Generated new TSK 0
I20260809 00:58:49.237284 13899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 00:58:49.244501 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13773
W20260809 00:58:49.245985 13863 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48469
W20260809 00:58:49.246784 10116 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13773: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 00:58:49.246965 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 13773
2026-08-09T00:58:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2401 ms)
[----------] 18 tests from AuthTokenIssuingTest (44549 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:49 dist-test-slave-j2vl krb5kdc[13912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:49 dist-test-slave-j2vl krb5kdc[13912](info): set up 2 sockets
Aug 09 00:58:49 dist-test-slave-j2vl krb5kdc[13912](info): commencing operation
krb5kdc: starting...
W20260809 00:58:50.652866 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.360s user 0.008s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:50 dist-test-slave-j2vl krb5kdc[13912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:58:51.037814 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:36653
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:36653,127.9.225.61:34225,127.9.225.60:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:51.428299 13932 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 00:58:51.428747 13932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:51.428862 13932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:51.429023 13932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:51.436806 13932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:51.436932 13932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:51.437039 13932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:51.437143 13932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:51.437258 13932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:51.447887 13932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:36653,127.9.225.61:34225,127.9.225.60:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:36653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:51.449522 13932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:51.451077 13932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:51.464972 13943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:51.465001 13940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:51.465083 13945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:51.465643 13932 server_base.cc:1034] running on GCE node
I20260809 00:58:51.466787 13932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:51.468438 13932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:51.469687 13932 hybrid_clock.cc:648] HybridClock initialized: now 1786237131469613 us; error 36 us; skew 500 ppm
I20260809 00:58:51.470103 13932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:51 dist-test-slave-j2vl krb5kdc[13912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237131, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:51.473763 13932 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:58:51.475672 13932 webserver.cc:460] Webserver started at http://127.9.225.62:43451/ using document root <none> and password file <none>
I20260809 00:58:51.476481 13932 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:51.476583 13932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:51.476902 13932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:51.480355 13932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "5c1e09fd70a74232abea20e653adfeba"
format_stamp: "Formatted at 2026-08-09 00:58:51 on dist-test-slave-j2vl"
server_key: "b551a7fc72d220f8ef4cd1d61c67d71a"
server_key_iv: "cb62e4b6bc2326830ab4a1f27a3810ac"
server_key_version: "encryptionkey@0"
I20260809 00:58:51.481232 13932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "5c1e09fd70a74232abea20e653adfeba"
format_stamp: "Formatted at 2026-08-09 00:58:51 on dist-test-slave-j2vl"
server_key: "b551a7fc72d220f8ef4cd1d61c67d71a"
server_key_iv: "cb62e4b6bc2326830ab4a1f27a3810ac"
server_key_version: "encryptionkey@0"
I20260809 00:58:51.487452 13932 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 00:58:51.491693 13957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:51.493301 13932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 00:58:51.493614 13932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "5c1e09fd70a74232abea20e653adfeba"
format_stamp: "Formatted at 2026-08-09 00:58:51 on dist-test-slave-j2vl"
server_key: "b551a7fc72d220f8ef4cd1d61c67d71a"
server_key_iv: "cb62e4b6bc2326830ab4a1f27a3810ac"
server_key_version: "encryptionkey@0"
I20260809 00:58:51.493882 13932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:51.528862 13932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:51.533990 13932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:51.534340 13932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:51.555608 13932 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:36653
I20260809 00:58:51.555630 14046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:36653 every 8 connection(s)
I20260809 00:58:51.557204 13932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:51.561722 14047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:51.567823 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 13932
I20260809 00:58:51.568101 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:51.568975 10116 external_mini_cluster.cc:1428] Setting key 9f7b8dd658f80ad2c566fbfc364dfd30
I20260809 00:58:51.570729 14047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:58:51 dist-test-slave-j2vl krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237131, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:58:51.601298 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:51.575556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.9.225.62:36653)
0809 00:58:51.576614 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:51.576632 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:51.577228 (+ 596us) server_negotiation.cc:407] Connection header received
0809 00:58:51.579553 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:51.579564 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:51.579948 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:51.580201 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:51.581574 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:51.583574 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:51.585163 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:51.586279 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:51.586507 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:51.592055 (+ 5548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:51.592066 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:51.592084 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:51.592125 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:51.595696 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:51.596615 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:51.596624 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:51.596626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:51.596808 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:51.597287 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:51.597291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:51.597293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:51.598094 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:51.598354 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:51.599050 (+ 696us) server_negotiation.cc:299] Negotiation successful
0809 00:58:51.599519 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":770,"thread_start_us":482,"threads_started":1}
I20260809 00:58:51.601356 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:51.575556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:36653 (local address 127.0.0.1:46652)
0809 00:58:51.577023 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:51.577072 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:51.579082 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:51.580236 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:51.580251 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:51.580901 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:51.581281 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:51.581301 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:51.583765 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:51.583771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:51.584936 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:51.584951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:51.586639 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:51.586650 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:51.586874 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:51.587677 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:51.587716 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:51.591651 (+ 3935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:51.595898 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:51.595913 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:51.595929 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:51.596439 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:51.596938 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:51.596945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:51.596948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:51.597140 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:51.598254 (+ 1114us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:51.598263 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:51.598532 (+ 269us) client_negotiation.cc:769] Sending connection context
0809 00:58:51.598886 (+ 354us) client_negotiation.cc:240] Negotiation successful
0809 00:58:51.599665 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":175,"spinlock_wait_cycles":32128,"thread_start_us":921,"threads_started":1}
I20260809 00:58:51.605098 14047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.225.61@KRBTEST.COM; defaulting to no policy
I20260809 00:58:51.606346 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:51.605904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55088)
0809 00:58:51.606084 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:51.606203 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 00:58:51.606667 13961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
I20260809 00:58:51.608560 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:51.608149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55090)
0809 00:58:51.608305 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:51.608389 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.61@KRBTEST.COM" created.
W20260809 00:58:51.608909 14047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:34225: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.61.keytab.
Entry for principal kudu/127.9.225.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.61.keytab.
I20260809 00:58:51.653672 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.61:34225
--webserver_interface=127.9.225.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.61.keytab
--principal=kudu/127.9.225.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:36653,127.9.225.61:34225,127.9.225.60:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:58:51.658341 14047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } attempt: 1
I20260809 00:58:51.659632 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:51.659124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55106)
0809 00:58:51.659349 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:51.659484 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 00:58:51.660820 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:51.660361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55108)
0809 00:58:51.660587 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:51.660663 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 00:58:51.661012 14047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:34225: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
I20260809 00:58:51.729291 14047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } attempt: 2
I20260809 00:58:51.730741 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:51.730277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55110)
0809 00:58:51.730489 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:51.730593 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260809 00:58:51.731851 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:51.731442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55124)
0809 00:58:51.731620 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:51.731694 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 00:58:51.732059 14047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:34225: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
I20260809 00:58:51.823370 14047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } attempt: 3
I20260809 00:58:51.824707 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:51.824229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55138)
0809 00:58:51.824460 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:51.824555 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260809 00:58:51.825855 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:51.825479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55146)
0809 00:58:51.825624 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:51.825701 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 00:58:51.826066 14047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:34225: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
I20260809 00:58:51.969447 14047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } attempt: 4
I20260809 00:58:51.970796 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:51.970226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55162)
0809 00:58:51.970568 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:51.970656 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260809 00:58:51.971859 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:51.971478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55168)
0809 00:58:51.971633 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:51.971727 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 00:58:51.972052 14047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:34225: Network error: Client connection negotiation failed: client connection to 127.9.225.61:34225: connect: Connection refused (error 111)
W20260809 00:58:52.049573 14058 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 00:58:52.050017 14058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:52.050148 14058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:52.050292 14058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:52.059552 14058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:52.059698 14058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:52.059813 14058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:52.059912 14058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:52.060029 14058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:52.071136 14058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:36653,127.9.225.61:34225,127.9.225.60:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.61.keytab
--principal=kudu/127.9.225.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.61:34225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.225.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:52.072773 14058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:52.074383 14058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:52.087874 14069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:52.088050 14065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:52.087874 14067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:52.089625 14058 server_base.cc:1034] running on GCE node
I20260809 00:58:52.090700 14058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:52.092494 14058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:52.093734 14058 hybrid_clock.cc:648] HybridClock initialized: now 1786237132093666 us; error 50 us; skew 500 ppm
I20260809 00:58:52.094162 14058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:52.103199 14058 init.cc:377] Logged in from keytab as kudu/127.9.225.61@KRBTEST.COM (short username kudu)
I20260809 00:58:52.105095 14058 webserver.cc:460] Webserver started at http://127.9.225.61:43555/ using document root <none> and password file <none>
I20260809 00:58:52.105994 14058 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:52.106122 14058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:52.106494 14058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:52.110086 14058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/data/instance:
uuid: "11357ade86af4ad99bb35850b7b8052f"
format_stamp: "Formatted at 2026-08-09 00:58:52 on dist-test-slave-j2vl"
server_key: "2f24af81dd4a78d3ecc90552eebdbfe0"
server_key_iv: "0051d10d2086684bd072d0bb6b3da4dd"
server_key_version: "encryptionkey@0"
I20260809 00:58:52.111114 14058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/wal/instance:
uuid: "11357ade86af4ad99bb35850b7b8052f"
format_stamp: "Formatted at 2026-08-09 00:58:52 on dist-test-slave-j2vl"
server_key: "2f24af81dd4a78d3ecc90552eebdbfe0"
server_key_iv: "0051d10d2086684bd072d0bb6b3da4dd"
server_key_version: "encryptionkey@0"
I20260809 00:58:52.117597 14058 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260809 00:58:52.121621 14080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:52.123236 14058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 00:58:52.123469 14058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/wal
uuid: "11357ade86af4ad99bb35850b7b8052f"
format_stamp: "Formatted at 2026-08-09 00:58:52 on dist-test-slave-j2vl"
server_key: "2f24af81dd4a78d3ecc90552eebdbfe0"
server_key_iv: "0051d10d2086684bd072d0bb6b3da4dd"
server_key_version: "encryptionkey@0"
I20260809 00:58:52.123728 14058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:52.149809 14058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:52.155507 14058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:52.155853 14058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:52.176780 14058 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.61:34225
I20260809 00:58:52.176795 14172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.61:34225 every 8 connection(s)
I20260809 00:58:52.178454 14058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
I20260809 00:58:52.182778 14173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:52.184774 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 14058
I20260809 00:58:52.184984 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-1/wal/instance
I20260809 00:58:52.185601 10116 external_mini_cluster.cc:1428] Setting key 050e85abf76052f9c6e32f78c49795ca
I20260809 00:58:52.193638 14173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.61@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:58:52.222127 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.197783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.9.225.62:36653)
0809 00:58:52.198488 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:52.198497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:52.199745 (+ 1248us) server_negotiation.cc:407] Connection header received
0809 00:58:52.202970 (+ 3225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:52.202979 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:52.203097 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:52.203244 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:52.205108 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.206515 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.208242 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.209040 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.209219 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.213620 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:52.213630 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:52.213638 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:52.213694 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:52.217918 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.218984 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.218994 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.219000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.219173 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.219662 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.219670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.219677 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.220162 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:52.220353 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:52.221244 (+ 891us) server_negotiation.cc:299] Negotiation successful
0809 00:58:52.221587 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":340,"threads_started":1}
I20260809 00:58:52.222946 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.198544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:36653 (local address 127.0.0.1:46668)
0809 00:58:52.199533 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.199598 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:52.202556 (+ 2958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:52.203502 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:52.203512 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:52.204158 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:52.204829 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.204849 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.206722 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.206726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.208068 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.208082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.209293 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.209299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.209611 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.210414 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:52.210441 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:52.213249 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:52.218147 (+ 4898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.218159 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.218178 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.218815 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.219302 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.219306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.219308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.219534 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.220341 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:52.220349 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:52.220805 (+ 456us) client_negotiation.cc:769] Sending connection context
0809 00:58:52.221081 (+ 276us) client_negotiation.cc:240] Negotiation successful
0809 00:58:52.221643 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":391,"threads_started":1}
WARNING: no policy specified for kudu/127.9.225.60@KRBTEST.COM; defaulting to no policy
I20260809 00:58:52.225087 14173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.60@KRBTEST.COM" created.
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.61@KRBTEST.COM for kudu/127.9.225.61@KRBTEST.COM
I20260809 00:58:52.242455 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.225909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55174)
0809 00:58:52.226095 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.226116 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:52.226228 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:52.227427 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:52.227434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:52.228000 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:52.228207 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.228222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.229841 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.229850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.231374 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.231394 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.232341 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.232353 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.232568 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.233273 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:52.233301 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:52.236101 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:52.239701 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.239709 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.239714 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.240112 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.240523 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.240531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.240534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.240687 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.241335 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:52.241391 (+ 56us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:52.241546 (+ 155us) client_negotiation.cc:769] Sending connection context
0809 00:58:52.241708 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 00:58:52.242040 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 00:58:52.242555 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.226170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.9.225.61:34225)
0809 00:58:52.226808 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:52.226817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:52.226916 (+ 99us) server_negotiation.cc:407] Connection header received
0809 00:58:52.227013 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:52.227020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:52.227259 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:52.227476 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:52.228414 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.229654 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.231532 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.232148 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.232252 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.236355 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:52.236383 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:52.236406 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:52.236442 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:52.239531 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.240267 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.240275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.240277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.240406 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.240794 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.240802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.240805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.241200 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:52.241513 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:52.241863 (+ 350us) server_negotiation.cc:299] Negotiation successful
0809 00:58:52.242164 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":481,"thread_start_us":342,"threads_started":1}
I20260809 00:58:52.245862 14173 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } has no permanent_uuid. Determining permanent_uuid...
I20260809 00:58:52.247424 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.246735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38566)
0809 00:58:52.246986 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.247183 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260809 00:58:52.247764 14086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:58:52.249895 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.249317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38582)
0809 00:58:52.249589 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.249711 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260809 00:58:52.250145 14173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:58:52.271315 14047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } attempt: 5
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.60.keytab.
Entry for principal kudu/127.9.225.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.60.keytab.
I20260809 00:58:52.274775 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.60:45565
--webserver_interface=127.9.225.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.60.keytab
--principal=kudu/127.9.225.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:36653,127.9.225.61:34225,127.9.225.60:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237131, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for kudu/127.9.225.61@KRBTEST.COM
I20260809 00:58:52.287858 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.272113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55178)
0809 00:58:52.272327 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.272348 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:52.272572 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:52.272942 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:52.272945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:52.273621 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:52.273903 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.273918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.275134 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.275138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.276036 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.276050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.277227 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.277231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.277458 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.278150 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:52.278169 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:52.280276 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:52.284994 (+ 4718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.284999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.285003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.285550 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.286024 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.286027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.286030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.286175 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.286915 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:52.286919 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:52.287131 (+ 212us) client_negotiation.cc:769] Sending connection context
0809 00:58:52.287317 (+ 186us) client_negotiation.cc:240] Negotiation successful
0809 00:58:52.287573 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 00:58:52.288208 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.272384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.9.225.61:34225)
0809 00:58:52.272578 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:52.272583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:52.272604 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:58:52.272710 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:52.272715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:52.272785 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:52.272885 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:52.274124 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.274870 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.276216 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.277051 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.277206 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.280514 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:52.280524 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:52.280532 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:52.280583 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:52.284762 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.285696 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.285705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.285711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.285895 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.286303 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.286311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.286316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.286764 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:52.286941 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:52.287496 (+ 555us) server_negotiation.cc:299] Negotiation successful
0809 00:58:52.287799 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 00:58:52.289377 14047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } has no permanent_uuid. Determining permanent_uuid...
I20260809 00:58:52.290659 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.290181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38588)
0809 00:58:52.290420 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.290503 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260809 00:58:52.291977 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.291520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38600)
0809 00:58:52.291737 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.291861 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260809 00:58:52.292232 14047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:58:52.299585 14173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 1
I20260809 00:58:52.301003 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.300488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38606)
0809 00:58:52.300719 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.300839 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260809 00:58:52.302207 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.301843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38618)
0809 00:58:52.302012 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.302095 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 00:58:52.302465 14173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:58:52.343538 14047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 1
I20260809 00:58:52.345292 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.344530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38630)
0809 00:58:52.345002 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.345110 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284}
I20260809 00:58:52.346390 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.346019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38634)
0809 00:58:52.346177 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.346262 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 00:58:52.347726 14047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:58:52.370705 14173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 2
I20260809 00:58:52.372071 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.371553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38638)
0809 00:58:52.371793 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.371896 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 00:58:52.373313 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.372910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38648)
0809 00:58:52.373082 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.373164 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260809 00:58:52.373719 14173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:58:52.416047 14047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 2
I20260809 00:58:52.417531 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.416853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38652)
0809 00:58:52.417107 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.417228 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 00:58:52.418872 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.418415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38656)
0809 00:58:52.418606 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.418703 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 00:58:52.419198 14047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:58:52.464970 14173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 3
I20260809 00:58:52.466578 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.465844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38666)
0809 00:58:52.466204 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.466328 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260809 00:58:52.468145 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.467501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38676)
0809 00:58:52.467859 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.467970 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260809 00:58:52.468428 14173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:58:52.525537 14047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 3
I20260809 00:58:52.526968 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.526292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38690)
0809 00:58:52.526579 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.526698 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 00:58:52.528399 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.527812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38694)
0809 00:58:52.528054 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.528178 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260809 00:58:52.528724 14047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:58:52.611707 14173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 4
I20260809 00:58:52.613128 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.612507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38698)
0809 00:58:52.612834 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.612942 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260809 00:58:52.614455 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.614044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38702)
0809 00:58:52.614227 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.614323 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260809 00:58:52.614760 14173 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
W20260809 00:58:52.671305 14186 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 00:58:52.671759 14186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:52.671911 14186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:52.672057 14186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:52.679926 14186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:52.680055 14186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:52.680154 14186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:52.680239 14186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:52.680356 14186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:52.691551 14186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:36653,127.9.225.61:34225,127.9.225.60:45565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.60.keytab
--principal=kudu/127.9.225.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.60:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.225.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:52.693168 14186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:52.694756 14186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 00:58:52.705996 14047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 4
I20260809 00:58:52.707839 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.707083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38706)
0809 00:58:52.707454 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.707576 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
W20260809 00:58:52.708945 14198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:52.709249 14196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:52.709546 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.708903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38720)
0809 00:58:52.709228 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.709333 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260809 00:58:52.709311 14202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:52.709861 14047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:45565: Network error: Client connection negotiation failed: client connection to 127.9.225.60:45565: connect: Connection refused (error 111)
I20260809 00:58:52.710649 14186 server_base.cc:1034] running on GCE node
I20260809 00:58:52.711660 14186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:52.713460 14186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:52.714771 14186 hybrid_clock.cc:648] HybridClock initialized: now 1786237132714666 us; error 68 us; skew 500 ppm
I20260809 00:58:52.715212 14186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:52.719386 14186 init.cc:377] Logged in from keytab as kudu/127.9.225.60@KRBTEST.COM (short username kudu)
I20260809 00:58:52.721235 14186 webserver.cc:460] Webserver started at http://127.9.225.60:35247/ using document root <none> and password file <none>
I20260809 00:58:52.722105 14186 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:52.722213 14186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:52.722590 14186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:52.726277 14186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/data/instance:
uuid: "9409f47bf2ce4706ac5167bffbb791ee"
format_stamp: "Formatted at 2026-08-09 00:58:52 on dist-test-slave-j2vl"
server_key: "b196e0da144532bdb3543b25aadb368d"
server_key_iv: "4894706eedab5641dab71223dacd8d93"
server_key_version: "encryptionkey@0"
I20260809 00:58:52.727377 14186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/wal/instance:
uuid: "9409f47bf2ce4706ac5167bffbb791ee"
format_stamp: "Formatted at 2026-08-09 00:58:52 on dist-test-slave-j2vl"
server_key: "b196e0da144532bdb3543b25aadb368d"
server_key_iv: "4894706eedab5641dab71223dacd8d93"
server_key_version: "encryptionkey@0"
I20260809 00:58:52.734225 14186 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 00:58:52.738708 14215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:52.740482 14186 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 00:58:52.740744 14186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/wal
uuid: "9409f47bf2ce4706ac5167bffbb791ee"
format_stamp: "Formatted at 2026-08-09 00:58:52 on dist-test-slave-j2vl"
server_key: "b196e0da144532bdb3543b25aadb368d"
server_key_iv: "4894706eedab5641dab71223dacd8d93"
server_key_version: "encryptionkey@0"
I20260809 00:58:52.741024 14186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:52.764454 14186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:52.769575 14186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:52.769923 14186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:52.791440 14186 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.60:45565
I20260809 00:58:52.791457 14294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.60:45565 every 8 connection(s)
I20260809 00:58:52.793136 14186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
I20260809 00:58:52.797581 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 14186
I20260809 00:58:52.797577 14295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:52.797834 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786236719337175-10116-0/minicluster-data/master-2/wal/instance
I20260809 00:58:52.798388 10116 external_mini_cluster.cc:1428] Setting key 9bbccaf03e6f1897997e110f80f11ca7
I20260809 00:58:52.809239 14295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.60@KRBTEST.COM
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.60@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:58:52.826004 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.800528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.9.225.60:45565)
0809 00:58:52.801535 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:52.801552 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:52.801610 (+ 58us) server_negotiation.cc:407] Connection header received
0809 00:58:52.803561 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:52.803600 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:52.804270 (+ 670us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:52.804859 (+ 589us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:52.806578 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.808602 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.810147 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.811186 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.811374 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.816493 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:52.816505 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:52.816522 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:52.816577 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:52.820273 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.821303 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.821311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.821319 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.821673 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.822117 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.822122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.822126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.822846 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:52.823140 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:52.823717 (+ 577us) server_negotiation.cc:299] Negotiation successful
0809 00:58:52.824289 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":386,"threads_started":1}
I20260809 00:58:52.830768 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.812463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:36653 (local address 127.0.0.1:46680)
0809 00:58:52.813525 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.813566 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:52.813775 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:52.814207 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:52.814214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:52.815111 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:52.815524 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.815541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.816768 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.816773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.818082 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.818097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.819228 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.819233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.819381 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.820097 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:52.820126 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:52.823020 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:52.827329 (+ 4309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.827342 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.827348 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.828023 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.828548 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.828553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.828556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.828765 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.829482 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:52.829488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:52.829726 (+ 238us) client_negotiation.cc:769] Sending connection context
0809 00:58:52.829953 (+ 227us) client_negotiation.cc:240] Negotiation successful
0809 00:58:52.830362 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":583,"threads_started":1}
I20260809 00:58:52.831086 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.812472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46680 (local address 127.9.225.62:36653)
0809 00:58:52.813133 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:52.813140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:52.813677 (+ 537us) server_negotiation.cc:407] Connection header received
0809 00:58:52.813931 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:52.813935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:52.814005 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:52.814134 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:52.815711 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.816568 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.818275 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.818993 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.819147 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.823279 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:52.823285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:52.823289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:52.823327 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:52.827132 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.828206 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.828211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.828214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.828376 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.828912 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.828917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.828919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.829306 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:52.829516 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:52.830121 (+ 605us) server_negotiation.cc:299] Negotiation successful
0809 00:58:52.830445 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":287,"threads_started":1}
I20260809 00:58:52.832648 14295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.60@KRBTEST.COM for kudu/127.9.225.61@KRBTEST.COM
I20260809 00:58:52.848646 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.829997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46696 (local address 127.9.225.62:36653)
0809 00:58:52.831361 (+ 1364us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:52.831366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:52.831385 (+ 19us) server_negotiation.cc:407] Connection header received
0809 00:58:52.831433 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:52.831438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:52.831558 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:52.831679 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:52.833286 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.834256 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.835638 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.836191 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.836288 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.841477 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:52.841484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:52.841487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:52.841528 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:52.844852 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.846043 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.846052 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.846056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.846257 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.846765 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.846773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.846776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.847231 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:52.847501 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:52.848001 (+ 500us) server_negotiation.cc:299] Negotiation successful
0809 00:58:52.848255 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1234,"thread_start_us":1474,"threads_started":1}
I20260809 00:58:52.849802 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.833571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:34225 (local address 127.0.0.1:55190)
0809 00:58:52.833762 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.833783 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:52.833899 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:52.835030 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:52.835035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:52.835496 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:52.835747 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.835759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.837109 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.837116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.838195 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.838210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.839320 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.839326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.839486 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.840138 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:52.840158 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:52.842557 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:52.846357 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.846363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.846366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.847126 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.847686 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.847691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.847694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.847943 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.848629 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:52.848637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:52.848835 (+ 198us) client_negotiation.cc:769] Sending connection context
0809 00:58:52.849037 (+ 202us) client_negotiation.cc:240] Negotiation successful
0809 00:58:52.849422 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 00:58:52.849912 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.833709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55190 (local address 127.9.225.61:34225)
0809 00:58:52.834694 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:52.834699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:52.834729 (+ 30us) server_negotiation.cc:407] Connection header received
0809 00:58:52.834779 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:52.834783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:52.834865 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:52.835001 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:52.835961 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.836925 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.838378 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.839114 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.839267 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.842798 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:52.842804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:52.842808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:52.842847 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:52.846155 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.847318 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.847326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.847329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.847528 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.848102 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.848108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.848112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.848490 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:52.848619 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:52.849145 (+ 526us) server_negotiation.cc:299] Negotiation successful
0809 00:58:52.849420 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":768,"threads_started":1}
I20260809 00:58:52.851008 14295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.60@KRBTEST.COM for kudu/127.9.225.60@KRBTEST.COM
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.61@KRBTEST.COM
I20260809 00:58:52.867626 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.851783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38742)
0809 00:58:52.852077 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.852095 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:52.852187 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:52.852529 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:52.852532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:52.852936 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:52.853100 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.853108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.854518 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.854526 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.855991 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.856007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.857096 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.857103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.857318 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.858133 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:52.858160 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:52.860948 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:52.864549 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.864555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.864559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.865125 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.865643 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.865649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.865654 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.865841 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.866481 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:52.866486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:52.866645 (+ 159us) client_negotiation.cc:769] Sending connection context
0809 00:58:52.866819 (+ 174us) client_negotiation.cc:240] Negotiation successful
0809 00:58:52.867182 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260809 00:58:52.867679 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.851974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.9.225.60:45565)
0809 00:58:52.852181 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:52.852189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:52.852210 (+ 21us) server_negotiation.cc:407] Connection header received
0809 00:58:52.852266 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:52.852270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:52.852355 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:52.852472 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:52.853254 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.854332 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.856180 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.856846 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.857006 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.861203 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:52.861210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:52.861215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:52.861255 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:52.864366 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.865275 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.865279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.865281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.865495 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.865986 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.865991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.865993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.866317 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:52.866516 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:52.866933 (+ 417us) server_negotiation.cc:299] Negotiation successful
0809 00:58:52.867361 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":173,"server-negotiator.queue_time_us":68}
I20260809 00:58:52.869467 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.851117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.9.225.61:34225)
0809 00:58:52.851321 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:52.851325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:52.851339 (+ 14us) server_negotiation.cc:407] Connection header received
0809 00:58:52.851380 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:52.851383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:52.851442 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:52.851569 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:52.853122 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.854347 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.856200 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.856915 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.857036 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.862519 (+ 5483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:52.862526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:52.862529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:52.862570 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:52.865909 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.866708 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.866714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.866718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.866884 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.867299 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.867307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.867311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.867602 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:52.867943 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:52.868769 (+ 826us) server_negotiation.cc:299] Negotiation successful
0809 00:58:52.869025 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 00:58:52.873798 14295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee: Bootstrap starting.
I20260809 00:58:52.875181 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:52.878366 14295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:52.879818 14295 log.cc:824] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:52.879390 14007 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5c1e09fd70a74232abea20e653adfeba"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46696
I20260809 00:58:52.883473 14295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee: No bootstrap required, opened a new log
I20260809 00:58:52.891963 14295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:58:52.892483 14295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9409f47bf2ce4706ac5167bffbb791ee, State: Initialized, Role: FOLLOWER
I20260809 00:58:52.893059 14295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:58:52.894282 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:58:52.894759 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:52.895720 14295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [sys.catalog]: configured and running, proceeding with master startup.
W20260809 00:58:52.907671 14321 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 00:58:52.907868 14321 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 00:58:52.904860 14186 master_runner.cc:186] Error getting master registration for 127.9.225.62:36653: OK, instance_id { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" instance_seqno: 1786237131549239 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 00:58:52.914175 14186 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 00:58:52.914050 14173 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 5
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237132, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.61@KRBTEST.COM for kudu/127.9.225.60@KRBTEST.COM
I20260809 00:58:52.931842 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.914864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38756)
0809 00:58:52.915122 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.915147 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:52.915280 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:52.915713 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:52.915719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:52.916301 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:52.916639 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.916663 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.918261 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.918268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.919659 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.919678 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.920828 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.920835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.921093 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.921983 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:52.922014 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:52.925031 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:52.928903 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.928927 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.928931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.929508 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.929948 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:52.929954 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:52.929958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:52.930107 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:52.930726 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:52.930731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:52.930904 (+ 173us) client_negotiation.cc:769] Sending connection context
0809 00:58:52.931053 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 00:58:52.931405 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 00:58:52.931844 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.914915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.9.225.60:45565)
0809 00:58:52.915226 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:52.915233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:52.915259 (+ 26us) server_negotiation.cc:407] Connection header received
0809 00:58:52.915439 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:52.915444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:52.915535 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:52.915656 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:52.916838 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.918082 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.919849 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.920624 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.920761 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.925296 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:52.925302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:52.925305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:52.925383 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:52.928691 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.929643 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.929650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.929654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.929819 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:52.930221 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:52.930227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:52.930231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:52.930597 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:52.930792 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:52.931180 (+ 388us) server_negotiation.cc:299] Negotiation successful
0809 00:58:52.931413 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260809 00:58:52.937891 14173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f: Bootstrap starting.
I20260809 00:58:52.942471 14173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:52.943712 14173 log.cc:824] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:52.947463 14173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f: No bootstrap required, opened a new log
I20260809 00:58:52.955726 14173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:58:52.956156 14173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11357ade86af4ad99bb35850b7b8052f, State: Initialized, Role: FOLLOWER
I20260809 00:58:52.956722 14173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:58:52.957911 14324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:58:52.958395 14324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:52.959331 14173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [sys.catalog]: configured and running, proceeding with master startup.
W20260809 00:58:52.971549 14342 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 00:58:52.971818 14342 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 00:58:52.968201 14058 master_runner.cc:186] Error getting master registration for 127.9.225.62:36653: OK, instance_id { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" instance_seqno: 1786237131549239 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 00:58:52.980722 14058 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 00:58:52.987154 14047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } attempt: 5
Aug 09 00:58:52 dist-test-slave-j2vl krb5kdc[13912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237131, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for kudu/127.9.225.60@KRBTEST.COM
I20260809 00:58:53.007663 14050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.988055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:45565 (local address 127.0.0.1:38770)
0809 00:58:52.988523 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:52.988547 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:52.988688 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:52.989479 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:52.989484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:52.990086 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:52.990328 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.990341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.991974 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.991981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.993581 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:52.993602 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.994781 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.994788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:52.995023 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.995849 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:52.995876 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:52.999430 (+ 3554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:53.004317 (+ 4887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:53.004326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:53.004330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:53.004936 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:53.005509 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:53.005516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:53.005520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:53.005752 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:53.006487 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:53.006493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:53.006685 (+ 192us) client_negotiation.cc:769] Sending connection context
0809 00:58:53.006855 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 00:58:53.007197 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315}
I20260809 00:58:53.007705 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:52.988289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.9.225.60:45565)
0809 00:58:52.988943 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:52.988950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:52.988969 (+ 19us) server_negotiation.cc:407] Connection header received
0809 00:58:52.989035 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:52.989041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:52.989121 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:52.989232 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:52.990551 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.991780 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.993759 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:52.994566 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:52.994742 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:52.999740 (+ 4998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:52.999748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:52.999754 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:52.999817 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:53.004104 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:53.005135 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:53.005142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:53.005146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:53.005382 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:53.005910 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:53.005916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:53.005922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:53.006332 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:53.006528 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:53.007016 (+ 488us) server_negotiation.cc:299] Negotiation successful
0809 00:58:53.007295 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511}
I20260809 00:58:53.015229 14047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba: Bootstrap starting.
I20260809 00:58:53.020074 14047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:53.021775 14047 log.cc:824] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba: Log is configured to *not* fsync() on all Append() calls
W20260809 00:58:53.022384 14223 tablet.cc:2369] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:58:53.032387 14047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba: No bootstrap required, opened a new log
I20260809 00:58:53.041316 14047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:58:53.041963 14047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1e09fd70a74232abea20e653adfeba, State: Initialized, Role: FOLLOWER
I20260809 00:58:53.042649 14047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:58:53.044190 14350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:58:53.045074 14350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:53.045635 14047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:53.045843 14007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 00:58:53.046190 14007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:53.050129 14007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:58:53.052991 14134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c1e09fd70a74232abea20e653adfeba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "11357ade86af4ad99bb35850b7b8052f"
I20260809 00:58:53.053627 14134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:53.057654 14134 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1e09fd70a74232abea20e653adfeba in term 1.
I20260809 00:58:53.058835 13961 leader_election.cc:302] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11357ade86af4ad99bb35850b7b8052f, 5c1e09fd70a74232abea20e653adfeba; no voters:
I20260809 00:58:53.061065 14259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c1e09fd70a74232abea20e653adfeba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9409f47bf2ce4706ac5167bffbb791ee"
I20260809 00:58:53.061709 14259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:53.066045 14259 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c1e09fd70a74232abea20e653adfeba in term 1.
I20260809 00:58:53.067103 14007 leader_election.cc:288] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [CANDIDATE]: Term 1 election: Requested vote from peers 11357ade86af4ad99bb35850b7b8052f (127.9.225.61:34225), 9409f47bf2ce4706ac5167bffbb791ee (127.9.225.60:45565)
I20260809 00:58:53.074666 14350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:53.088639 14350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [term 1 LEADER]: Becoming Leader. State: Replica: 5c1e09fd70a74232abea20e653adfeba, State: Running, Role: LEADER
I20260809 00:58:53.089610 14350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } }
I20260809 00:58:53.109242 14367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c1e09fd70a74232abea20e653adfeba. Latest consensus state: current_term: 1 leader_uuid: "5c1e09fd70a74232abea20e653adfeba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:58:53.109959 14367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:53.121384 14379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:53.128484 14382 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 00:58:53.149330 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:53.117538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.9.225.60:45565)
0809 00:58:53.117758 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.117764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.122264 (+ 4500us) server_negotiation.cc:407] Connection header received
0809 00:58:53.122591 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.122599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.122699 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.122785 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.129216 (+ 6431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:53.130352 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:53.132348 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:53.133095 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:53.133232 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:53.140353 (+ 7121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:53.140360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:53.140365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:53.140414 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:53.144956 (+ 4542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:53.146232 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:53.146244 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:53.146248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:53.146433 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:53.147107 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:53.147114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:53.147117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:53.147631 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:53.147931 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:53.148570 (+ 639us) server_negotiation.cc:299] Negotiation successful
0809 00:58:53.148886 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260809 00:58:53.159091 14089 tablet.cc:2369] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:58:53.169394 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:53.111116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.9.225.61:34225)
0809 00:58:53.111323 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.111330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.116855 (+ 5525us) server_negotiation.cc:407] Connection header received
0809 00:58:53.121825 (+ 4970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.121834 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.121957 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.122048 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.132934 (+ 10886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:53.134056 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:53.142761 (+ 8705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:53.143554 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:53.143681 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:53.155721 (+ 12040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:53.155731 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:53.155736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:53.155792 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:53.164689 (+ 8897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:53.165924 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:53.165933 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:53.165939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:53.166139 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:53.166644 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:53.166651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:53.166654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:53.167111 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:53.167410 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:53.168627 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0809 00:58:53.168919 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 00:58:53.170082 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:53.099454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46702 (local address 127.9.225.62:36653)
0809 00:58:53.099666 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.099677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.105166 (+ 5489us) server_negotiation.cc:407] Connection header received
0809 00:58:53.117936 (+ 12770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.117944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.118043 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.118124 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.126623 (+ 8499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:53.127958 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:53.140897 (+ 12939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:53.141658 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:53.141768 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:53.157023 (+ 15255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:53.157032 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:53.157037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:53.157110 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:53.165314 (+ 8204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:53.166318 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:53.166326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:53.166330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:53.166525 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:53.167018 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:53.167024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:53.167028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:53.167438 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:53.167706 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:53.168974 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0809 00:58:53.169289 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 00:58:53.171720 14379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 00:58:53.196231 14259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [term 1 FOLLOWER]: Refusing update from remote peer 5c1e09fd70a74232abea20e653adfeba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:58:53.197736 14134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [term 1 FOLLOWER]: Refusing update from remote peer 5c1e09fd70a74232abea20e653adfeba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:58:53.199258 14350 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [LEADER]: Connected to new peer: Peer: permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:58:53.201073 14350 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:58:53.230543 14324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c1e09fd70a74232abea20e653adfeba. Latest consensus state: current_term: 1 leader_uuid: "5c1e09fd70a74232abea20e653adfeba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:58:53.230952 14324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:53.239041 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c1e09fd70a74232abea20e653adfeba. Latest consensus state: current_term: 1 leader_uuid: "5c1e09fd70a74232abea20e653adfeba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:58:53.239459 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:53.239759 14387 mvcc.cc:205] Tried to move back new op lower bound from 7316427297555931136 to 7316427297155710976. Current Snapshot: MvccSnapshot[applied={T|T < 7316427297555931136}]
I20260809 00:58:53.246183 14389 mvcc.cc:205] Tried to move back new op lower bound from 7316427297555931136 to 7316427297155710976. Current Snapshot: MvccSnapshot[applied={T|T < 7316427297555931136}]
I20260809 00:58:53.278240 14308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c1e09fd70a74232abea20e653adfeba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:58:53.278590 14308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:53.280615 14324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c1e09fd70a74232abea20e653adfeba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:58:53.280923 14324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:53.287752 14367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c1e09fd70a74232abea20e653adfeba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:58:53.288115 14367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:53.288499 14367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c1e09fd70a74232abea20e653adfeba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c1e09fd70a74232abea20e653adfeba" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 36653 } } peers { permanent_uuid: "11357ade86af4ad99bb35850b7b8052f" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 34225 } } peers { permanent_uuid: "9409f47bf2ce4706ac5167bffbb791ee" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 45565 } } }
I20260809 00:58:53.288723 14367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:53.290614 14379 catalog_manager.cc:1347] Generated new cluster ID: 7f018e7155374b2b9cf0e77aef759d0a
I20260809 00:58:53.290720 14379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:53.323181 14379 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:53.325179 14379 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:53.361187 14379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c1e09fd70a74232abea20e653adfeba: Generated new TSK 0
I20260809 00:58:53.362797 14379 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 00:58:53.536520 14300 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:53.524549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.9.225.62:36653)
0809 00:58:53.524826 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.524833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.531840 (+ 7007us) server_negotiation.cc:407] Connection header received
0809 00:58:53.533461 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.533475 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.533579 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.533662 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.536211 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46716: BlockingRecv error: recv got EOF from 127.0.0.1:46716 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260809 00:58:53.538988 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:53.532615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.9.225.61:34225)
0809 00:58:53.532853 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.532859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.536233 (+ 3374us) server_negotiation.cc:407] Connection header received
0809 00:58:53.537302 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.537309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.537445 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.537587 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.538715 (+ 1128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55214: BlockingRecv error: recv got EOF from 127.0.0.1:55214 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 00:58:53.542743 14299 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:53.540703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.9.225.60:45565)
0809 00:58:53.540882 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.540887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.541003 (+ 116us) server_negotiation.cc:407] Connection header received
0809 00:58:53.541182 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.541189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.541282 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.541447 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.542469 (+ 1022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38792: BlockingRecv error: recv got EOF from 127.0.0.1:38792 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260809 00:58:53.568046 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:53.555638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.9.225.61:34225)
0809 00:58:53.555894 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.555901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.565527 (+ 9626us) server_negotiation.cc:407] Connection header received
0809 00:58:53.566163 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.566170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.566332 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.566420 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.567745 (+ 1325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55230: BlockingRecv error: recv got EOF from 127.0.0.1:55230 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260809 00:58:53.591282 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:53.588995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.9.225.61:34225)
0809 00:58:53.589206 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.589212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.589511 (+ 299us) server_negotiation.cc:407] Connection header received
0809 00:58:53.589736 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.589742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.589862 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.589990 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.591000 (+ 1010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55232: BlockingRecv error: recv got EOF from 127.0.0.1:55232 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260809 00:58:53.603813 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:53.597485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.9.225.61:34225)
0809 00:58:53.597698 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.597707 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.602078 (+ 4371us) server_negotiation.cc:407] Connection header received
0809 00:58:53.602292 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.602298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.602409 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.602545 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.603501 (+ 956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55246: BlockingRecv error: recv got EOF from 127.0.0.1:55246 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260809 00:58:53.615206 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:53.612530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.9.225.61:34225)
0809 00:58:53.612817 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.612824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.613394 (+ 570us) server_negotiation.cc:407] Connection header received
0809 00:58:53.613625 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.613631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.613753 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.613873 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.614918 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55248: BlockingRecv error: recv got EOF from 127.0.0.1:55248 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260809 00:58:53.630450 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:53.627897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.9.225.61:34225)
0809 00:58:53.628161 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.628168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.628612 (+ 444us) server_negotiation.cc:407] Connection header received
0809 00:58:53.628902 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.628907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.629054 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.629174 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.630171 (+ 997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55258: BlockingRecv error: recv got EOF from 127.0.0.1:55258 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260809 00:58:53.653213 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:53.651108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.9.225.61:34225)
0809 00:58:53.651330 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.651336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.651705 (+ 369us) server_negotiation.cc:407] Connection header received
0809 00:58:53.651889 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.651895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.652008 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.652139 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.652944 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55262: BlockingRecv error: recv got EOF from 127.0.0.1:55262 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260809 00:58:53.697883 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:53.695530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55264 (local address 127.9.225.61:34225)
0809 00:58:53.695780 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.695790 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.696205 (+ 415us) server_negotiation.cc:407] Connection header received
0809 00:58:53.696450 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.696454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.696635 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.696754 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.697614 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55264: BlockingRecv error: recv got EOF from 127.0.0.1:55264 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 00:58:53.768594 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:53.766185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.9.225.61:34225)
0809 00:58:53.766435 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.766441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.766834 (+ 393us) server_negotiation.cc:407] Connection header received
0809 00:58:53.767099 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.767104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.767207 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.767343 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.768336 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55278: BlockingRecv error: recv got EOF from 127.0.0.1:55278 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260809 00:58:53.903332 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:53.900980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.9.225.61:34225)
0809 00:58:53.901271 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:53.901278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:53.901625 (+ 347us) server_negotiation.cc:407] Connection header received
0809 00:58:53.901909 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:53.901913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:53.902001 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:53.902125 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:53.903036 (+ 911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55282: BlockingRecv error: recv got EOF from 127.0.0.1:55282 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
I20260809 00:58:53.910030 14321 catalog_manager.cc:1259] Loaded cluster ID: 7f018e7155374b2b9cf0e77aef759d0a
I20260809 00:58:53.910198 14321 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee: loading cluster ID for follower catalog manager: success
I20260809 00:58:53.912962 14321 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee: acquiring CA information for follower catalog manager: success
I20260809 00:58:53.915190 14321 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9409f47bf2ce4706ac5167bffbb791ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 00:58:53.974177 14342 catalog_manager.cc:1259] Loaded cluster ID: 7f018e7155374b2b9cf0e77aef759d0a
I20260809 00:58:53.974298 14342 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f: loading cluster ID for follower catalog manager: success
I20260809 00:58:53.977303 14342 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f: acquiring CA information for follower catalog manager: success
I20260809 00:58:53.979470 14342 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 11357ade86af4ad99bb35850b7b8052f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 00:58:54.170883 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:54.164011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.9.225.61:34225)
0809 00:58:54.164303 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:54.164310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:54.164635 (+ 325us) server_negotiation.cc:407] Connection header received
0809 00:58:54.164904 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:54.164909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:54.165096 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:54.165236 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:58:54.165785 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:54.166718 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:54.168231 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:54.168668 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:54.168826 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:54.169482 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:58:54.170105 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:58:54.170313 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:54.170456 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 00:58:54.170566 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260809 00:58:54.189018 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:54.181620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.9.225.60:45565)
0809 00:58:54.182460 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:54.182465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:54.182485 (+ 20us) server_negotiation.cc:407] Connection header received
0809 00:58:54.182597 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:54.182601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:54.182775 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:54.182942 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:58:54.183396 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:54.184194 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:54.186328 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:54.186820 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:54.186948 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:54.187618 (+ 670us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:58:54.188253 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:58:54.188373 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:54.188523 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 00:58:54.188604 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":610,"threads_started":1}
I20260809 00:58:54.205643 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:54.197126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.9.225.62:36653)
0809 00:58:54.198065 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:54.198073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:54.198102 (+ 29us) server_negotiation.cc:407] Connection header received
0809 00:58:54.198237 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:54.198241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:54.198510 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:54.198722 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:58:54.199797 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:54.200724 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:54.202850 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:54.203558 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:54.203721 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:54.204400 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:58:54.204890 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:58:54.205014 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:54.205181 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 00:58:54.205293 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":559,"threads_started":1}
I20260809 00:58:54.205672 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:54.197872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.9.225.61:34225)
0809 00:58:54.198086 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:54.198094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:54.198984 (+ 890us) server_negotiation.cc:407] Connection header received
0809 00:58:54.199364 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:54.199369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:54.199445 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:54.199552 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:58:54.200288 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:54.201469 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:54.203412 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:54.204039 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:54.204200 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:54.204806 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:58:54.204927 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:58:54.205013 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:54.205273 (+ 260us) server_negotiation.cc:299] Negotiation successful
0809 00:58:54.205400 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 00:58:54.207101 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:54.199222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.9.225.60:45565)
0809 00:58:54.199429 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:54.199436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:54.200306 (+ 870us) server_negotiation.cc:407] Connection header received
0809 00:58:54.200589 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:54.200596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:54.200689 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:54.200754 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:58:54.201716 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:54.202428 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:54.204720 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:54.205346 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:54.205477 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:54.206169 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:58:54.206318 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:58:54.206387 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:54.206635 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 00:58:54.206711 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 00:58:54.212960 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 13932
I20260809 00:58:54.214255 14040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:54.330112 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 13932
I20260809 00:58:54.371640 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 14058
I20260809 00:58:54.372991 14163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:54.490430 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 14058
I20260809 00:58:54.519738 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 14186
I20260809 00:58:54.521140 14288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:54.635135 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 14186
2026-08-09T00:58:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5396 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 00:58:54 dist-test-slave-j2vl krb5kdc[14517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 00:58:54 dist-test-slave-j2vl krb5kdc[14517](info): set up 2 sockets
Aug 09 00:58:54 dist-test-slave-j2vl krb5kdc[14517](info): commencing operation
krb5kdc: starting...
W20260809 00:58:56.063179 10116 mini_kdc.cc:121] Time spent starting KDC: real 1.374s user 0.003s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/test-user.keytab.
Aug 09 00:58:56 dist-test-slave-j2vl krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T00:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T00:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.225.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
Entry for principal kudu/127.9.225.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab.
I20260809 00:58:56.470026 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:41643
--webserver_interface=127.9.225.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:41643,127.9.225.61:43995,127.9.225.60:46549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 00:58:56.862375 14534 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 00:58:56.862789 14534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:56.862988 14534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:56.863142 14534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:56.871505 14534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:56.871639 14534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:56.871744 14534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:56.871861 14534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:56.871978 14534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:56.883455 14534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:41643,127.9.225.61:43995,127.9.225.60:46549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.62.keytab
--principal=kudu/127.9.225.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.62:41643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.225.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:56.885116 14534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:56.886725 14534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:56.901206 14548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:56.901266 14543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:56.901386 14544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:56.902462 14534 server_base.cc:1034] running on GCE node
I20260809 00:58:56.903469 14534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:56.905180 14534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:56.906498 14534 hybrid_clock.cc:648] HybridClock initialized: now 1786237136906461 us; error 77 us; skew 500 ppm
I20260809 00:58:56.906955 14534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:56 dist-test-slave-j2vl krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237136, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:56.910717 14534 init.cc:377] Logged in from keytab as kudu/127.9.225.62@KRBTEST.COM (short username kudu)
I20260809 00:58:56.912488 14534 webserver.cc:460] Webserver started at http://127.9.225.62:36423/ using document root <none> and password file <none>
I20260809 00:58:56.913389 14534 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:56.913547 14534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:56.913923 14534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:56.917558 14534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/data/instance:
uuid: "2f7d1ec733a94886b93d7d89b68e2074"
format_stamp: "Formatted at 2026-08-09 00:58:56 on dist-test-slave-j2vl"
server_key: "4b6e6695b094e314eb37d15c49f81204"
server_key_iv: "99db677b6acfdb8debca728577377982"
server_key_version: "encryptionkey@0"
I20260809 00:58:56.918529 14534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/wal/instance:
uuid: "2f7d1ec733a94886b93d7d89b68e2074"
format_stamp: "Formatted at 2026-08-09 00:58:56 on dist-test-slave-j2vl"
server_key: "4b6e6695b094e314eb37d15c49f81204"
server_key_iv: "99db677b6acfdb8debca728577377982"
server_key_version: "encryptionkey@0"
I20260809 00:58:56.924758 14534 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 00:58:56.930032 14561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:56.932155 14534 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260809 00:58:56.932447 14534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/wal
uuid: "2f7d1ec733a94886b93d7d89b68e2074"
format_stamp: "Formatted at 2026-08-09 00:58:56 on dist-test-slave-j2vl"
server_key: "4b6e6695b094e314eb37d15c49f81204"
server_key_iv: "99db677b6acfdb8debca728577377982"
server_key_version: "encryptionkey@0"
I20260809 00:58:56.932765 14534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:56.950142 14534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:56.958125 14534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:56.958531 14534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:56.980065 14534 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.62:41643
I20260809 00:58:56.980093 14642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.62:41643 every 8 connection(s)
I20260809 00:58:56.982002 14534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/data/info.pb
I20260809 00:58:56.987402 14643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:56.991041 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 14534
I20260809 00:58:56.991281 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-0/wal/instance
I20260809 00:58:56.991834 10116 external_mini_cluster.cc:1428] Setting key 61444cbf9abec93ec11dfb7663d2382e
I20260809 00:58:56.995858 14643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:58:57 dist-test-slave-j2vl krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237136, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:58:57.022104 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:56.999951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:41643 (local address 127.0.0.1:46280)
0809 00:58:57.001491 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.001555 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:57.003597 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:57.004780 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:57.004786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:57.005529 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:57.005921 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:57.005940 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.008030 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.008036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:57.009179 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:57.009193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.010243 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.010251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:57.010516 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:57.011274 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:57.011306 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:57.013754 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:57.017384 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:57.017397 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:57.017412 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:57.017847 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:57.018333 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:57.018339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:57.018343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:57.018522 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:57.019365 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:57.019379 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:57.019628 (+ 249us) client_negotiation.cc:769] Sending connection context
0809 00:58:57.019895 (+ 267us) client_negotiation.cc:240] Negotiation successful
0809 00:58:57.020592 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"mutex_wait_us":206,"spinlock_wait_cycles":21888,"thread_start_us":928,"threads_started":1}
I20260809 00:58:57.022140 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:56.999951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.9.225.62:41643)
0809 00:58:57.001028 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:57.001049 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:57.001715 (+ 666us) server_negotiation.cc:407] Connection header received
0809 00:58:57.004128 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:57.004140 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:57.004507 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:57.004756 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:57.006176 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.007846 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.009374 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.009983 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.010205 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:57.014020 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:57.014029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:57.014044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:57.014080 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:57.017179 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:57.018011 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:57.018018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:57.018020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:57.018214 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:57.018645 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:57.018652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:57.018660 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:57.019219 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:57.019458 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:57.020030 (+ 572us) server_negotiation.cc:299] Negotiation successful
0809 00:58:57.020463 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":759,"thread_start_us":456,"threads_started":1}
I20260809 00:58:57.025859 14643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.225.61@KRBTEST.COM; defaulting to no policy
I20260809 00:58:57.027498 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.026834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35774)
0809 00:58:57.027065 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.027328 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260809 00:58:57.027825 14571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.61@KRBTEST.COM" created.
I20260809 00:58:57.029976 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.029542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35780)
0809 00:58:57.029715 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.029802 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 00:58:57.030236 14643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:43995: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.61.keytab.
Entry for principal kudu/127.9.225.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.61.keytab.
I20260809 00:58:57.073731 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.61:43995
--webserver_interface=127.9.225.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.61.keytab
--principal=kudu/127.9.225.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:41643,127.9.225.61:43995,127.9.225.60:46549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:58:57.079726 14643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } attempt: 1
I20260809 00:58:57.081147 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.080595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35788)
0809 00:58:57.080865 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.081003 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 00:58:57.082320 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.081895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35792)
0809 00:58:57.082107 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.082184 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 00:58:57.082528 14643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:43995: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
I20260809 00:58:57.150827 14643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } attempt: 2
I20260809 00:58:57.152196 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.151662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35804)
0809 00:58:57.151930 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.152045 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 00:58:57.153513 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.153038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35820)
0809 00:58:57.153199 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.153279 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 00:58:57.153837 14643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:43995: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
I20260809 00:58:57.245146 14643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } attempt: 3
I20260809 00:58:57.246491 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.245990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35832)
0809 00:58:57.246220 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.246333 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 00:58:57.247629 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.247273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35836)
0809 00:58:57.247409 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.247484 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 00:58:57.247924 14643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:43995: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
I20260809 00:58:57.391290 14643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } attempt: 4
I20260809 00:58:57.392728 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.392113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35844)
0809 00:58:57.392485 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.392586 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260809 00:58:57.394032 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.393626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35852)
0809 00:58:57.393785 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.393867 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 00:58:57.394349 14643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.61:43995: Network error: Client connection negotiation failed: client connection to 127.9.225.61:43995: connect: Connection refused (error 111)
W20260809 00:58:57.461355 14655 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 00:58:57.461886 14655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:57.461977 14655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:57.462139 14655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:57.471120 14655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:57.471247 14655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:57.471354 14655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:57.471444 14655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:57.471544 14655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:57.483009 14655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:41643,127.9.225.61:43995,127.9.225.60:46549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.61.keytab
--principal=kudu/127.9.225.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.61:43995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.225.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:57.484675 14655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:57.486231 14655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:57.500269 14668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:57.500293 14664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:57.500209 14666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:57.501775 14655 server_base.cc:1034] running on GCE node
I20260809 00:58:57.502831 14655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:57.504612 14655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:57.505946 14655 hybrid_clock.cc:648] HybridClock initialized: now 1786237137505840 us; error 82 us; skew 500 ppm
I20260809 00:58:57.506381 14655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:57 dist-test-slave-j2vl krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237137, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:57.514784 14655 init.cc:377] Logged in from keytab as kudu/127.9.225.61@KRBTEST.COM (short username kudu)
I20260809 00:58:57.516773 14655 webserver.cc:460] Webserver started at http://127.9.225.61:34549/ using document root <none> and password file <none>
I20260809 00:58:57.517761 14655 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:57.517876 14655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:57.518263 14655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 00:58:57.522202 14655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/data/instance:
uuid: "48652feb3e97483ea89959893a6ce717"
format_stamp: "Formatted at 2026-08-09 00:58:57 on dist-test-slave-j2vl"
server_key: "a8e9b7eb1f86b76a7fbb2c885269a68d"
server_key_iv: "ae32ea6b53ddad08a123c61e513d7245"
server_key_version: "encryptionkey@0"
I20260809 00:58:57.523248 14655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/wal/instance:
uuid: "48652feb3e97483ea89959893a6ce717"
format_stamp: "Formatted at 2026-08-09 00:58:57 on dist-test-slave-j2vl"
server_key: "a8e9b7eb1f86b76a7fbb2c885269a68d"
server_key_iv: "ae32ea6b53ddad08a123c61e513d7245"
server_key_version: "encryptionkey@0"
I20260809 00:58:57.529698 14655 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260809 00:58:57.533934 14679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:57.535497 14655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 00:58:57.535722 14655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/wal
uuid: "48652feb3e97483ea89959893a6ce717"
format_stamp: "Formatted at 2026-08-09 00:58:57 on dist-test-slave-j2vl"
server_key: "a8e9b7eb1f86b76a7fbb2c885269a68d"
server_key_iv: "ae32ea6b53ddad08a123c61e513d7245"
server_key_version: "encryptionkey@0"
I20260809 00:58:57.535960 14655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:57.564304 14655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:57.570104 14655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:57.570458 14655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:57.591089 14655 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.61:43995
I20260809 00:58:57.591109 14755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.61:43995 every 8 connection(s)
I20260809 00:58:57.592631 14655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/data/info.pb
I20260809 00:58:57.593681 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 14655
I20260809 00:58:57.593948 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-1/wal/instance
I20260809 00:58:57.594453 10116 external_mini_cluster.cc:1428] Setting key 82c39dc135ac9d40559106a278438ca7
I20260809 00:58:57.597810 14757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:57.605304 14757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:58:57 dist-test-slave-j2vl krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237137, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.61@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.225.60@KRBTEST.COM; defaulting to no policy
I20260809 00:58:57.629539 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.608552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.9.225.62:41643)
0809 00:58:57.609177 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:57.609182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:57.610366 (+ 1184us) server_negotiation.cc:407] Connection header received
0809 00:58:57.612572 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:57.612578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:57.612688 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:57.612818 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:57.614880 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.615694 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.617598 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.618094 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.618269 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:57.622493 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:57.622499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:57.622503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:57.622555 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:57.625706 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:57.626704 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:57.626708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:57.626711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:57.626854 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:57.627340 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:57.627349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:57.627352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:57.627655 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:57.627831 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:57.628809 (+ 978us) server_negotiation.cc:299] Negotiation successful
0809 00:58:57.629174 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":370,"threads_started":1}
I20260809 00:58:57.630824 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.609226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:41643 (local address 127.0.0.1:46286)
0809 00:58:57.610180 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.610223 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:57.612201 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:57.613133 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:57.613143 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:57.613876 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:57.614583 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:57.614605 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.615910 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.615915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:57.617266 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:57.617281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.618404 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.618412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:57.618712 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:57.619551 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:57.619578 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:57.622127 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:57.625923 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:57.625935 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:57.625966 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:57.626536 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:57.626981 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:57.626986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:57.626990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:57.627205 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:57.627853 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:57.627866 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:57.628359 (+ 493us) client_negotiation.cc:769] Sending connection context
0809 00:58:57.628649 (+ 290us) client_negotiation.cc:240] Negotiation successful
0809 00:58:57.629209 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":392,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.225.60@KRBTEST.COM" created.
I20260809 00:58:57.633417 14757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:58:57 dist-test-slave-j2vl krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237137, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.61@KRBTEST.COM for kudu/127.9.225.61@KRBTEST.COM
I20260809 00:58:57.651870 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.634782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.9.225.61:43995)
0809 00:58:57.635566 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:57.635573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:57.635601 (+ 28us) server_negotiation.cc:407] Connection header received
0809 00:58:57.635649 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:57.635655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:57.635867 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:57.636035 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:57.636870 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.638145 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.639432 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.640001 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.640135 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:57.643863 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:57.643883 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:57.643909 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:57.643972 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:57.648017 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:57.649069 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:57.649077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:57.649080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:57.649210 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:57.649898 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:57.649902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:57.649904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:57.650438 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:57.650731 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:57.651244 (+ 513us) server_negotiation.cc:299] Negotiation successful
0809 00:58:57.651548 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":621,"thread_start_us":480,"threads_started":1}
I20260809 00:58:57.651921 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.634435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35858)
0809 00:58:57.634626 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.634650 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:57.634812 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:57.636024 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:57.636029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:57.636542 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:57.636676 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:57.636685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.638293 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.638299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:57.639247 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:57.639259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.640157 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.640163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:57.640368 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:57.641121 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:57.641139 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:57.643601 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:57.648221 (+ 4620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:57.648230 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:57.648237 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:57.648893 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:57.649507 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:57.649523 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:57.649529 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:57.649759 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:57.650619 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:57.650627 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:57.650846 (+ 219us) client_negotiation.cc:769] Sending connection context
0809 00:58:57.651020 (+ 174us) client_negotiation.cc:240] Negotiation successful
0809 00:58:57.651405 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 00:58:57.655104 14757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } has no permanent_uuid. Determining permanent_uuid...
I20260809 00:58:57.656589 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.655929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39576)
0809 00:58:57.656205 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.656402 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260809 00:58:57.657011 14685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
I20260809 00:58:57.659152 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.658719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39590)
0809 00:58:57.658919 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.659016 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260809 00:58:57.659472 14757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:46549: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.225.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.60.keytab.
Entry for principal kudu/127.9.225.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.60.keytab.
I20260809 00:58:57.678031 10116 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskizkHom/build/asan/bin/kudu
/tmp/dist-test-taskizkHom/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.60:46549
--webserver_interface=127.9.225.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.60.keytab
--principal=kudu/127.9.225.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.225.20:39183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.225.62:41643,127.9.225.61:43995,127.9.225.60:46549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 00:58:57.693638 14643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } attempt: 5
Aug 09 00:58:57 dist-test-slave-j2vl krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237136, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for kudu/127.9.225.61@KRBTEST.COM
I20260809 00:58:57.708907 14757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } attempt: 1
I20260809 00:58:57.710246 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.709657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39592)
0809 00:58:57.709910 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.710038 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 00:58:57.711091 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.694541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.9.225.61:43995)
0809 00:58:57.694748 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:57.694753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:57.694843 (+ 90us) server_negotiation.cc:407] Connection header received
0809 00:58:57.695071 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:57.695075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:57.695145 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:57.695236 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:57.696440 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.697185 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.699550 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.700076 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.700199 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:57.704631 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:57.704637 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:57.704640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:57.704682 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:57.707761 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:57.708634 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:57.708647 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:57.708650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:57.708796 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:57.709312 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:57.709316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:57.709318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:57.709812 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:57.709994 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:57.710479 (+ 485us) server_negotiation.cc:299] Negotiation successful
0809 00:58:57.710754 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 00:58:57.711506 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.711014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39594)
0809 00:58:57.711219 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.711337 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 00:58:57.711670 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.694435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35868)
0809 00:58:57.694707 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.694747 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:57.694924 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:57.695318 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:57.695326 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:57.695975 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:57.696253 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:57.696270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.697398 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.697407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:57.699060 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:57.699093 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:57.700286 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:57.700295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:57.700546 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:57.701429 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:57.701474 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:57.704345 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:57.707970 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:57.707976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:57.707981 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:57.708483 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:57.708934 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:57.708939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:57.708943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:57.709099 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:57.709950 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:57.709955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:57.710140 (+ 185us) client_negotiation.cc:769] Sending connection context
0809 00:58:57.710336 (+ 196us) client_negotiation.cc:240] Negotiation successful
0809 00:58:57.711294 (+ 958us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
W20260809 00:58:57.711818 14757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:46549: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
I20260809 00:58:57.712841 14643 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } has no permanent_uuid. Determining permanent_uuid...
I20260809 00:58:57.713977 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.713569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39606)
0809 00:58:57.713742 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.713826 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 00:58:57.715000 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.714612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39608)
0809 00:58:57.714803 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.714896 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 00:58:57.715238 14643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:46549: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
I20260809 00:58:57.766505 14643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } attempt: 1
I20260809 00:58:57.767946 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.767312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39616)
0809 00:58:57.767577 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.767712 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 00:58:57.769199 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.768773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39628)
0809 00:58:57.768951 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.769051 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 00:58:57.769536 14643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:46549: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
I20260809 00:58:57.780084 14757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } attempt: 2
I20260809 00:58:57.781584 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.780928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39644)
0809 00:58:57.781202 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.781368 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 00:58:57.782879 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.782440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39658)
0809 00:58:57.782608 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.782703 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 00:58:57.783182 14757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:46549: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
I20260809 00:58:57.837788 14643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } attempt: 2
I20260809 00:58:57.839388 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.838644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39674)
0809 00:58:57.838916 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.839040 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 00:58:57.840718 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.840257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39680)
0809 00:58:57.840457 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.840553 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260809 00:58:57.841039 14643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:46549: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
I20260809 00:58:57.874480 14757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } attempt: 3
I20260809 00:58:57.875849 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.875273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39688)
0809 00:58:57.875531 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.875644 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 00:58:57.877192 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.876574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39692)
0809 00:58:57.876877 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.877000 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260809 00:58:57.877537 14757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:46549: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
I20260809 00:58:57.947365 14643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } attempt: 3
I20260809 00:58:57.948890 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.948277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39700)
0809 00:58:57.948541 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.948658 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260809 00:58:57.950235 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:57.949759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39716)
0809 00:58:57.950009 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:57.950099 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260809 00:58:57.950606 14643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:46549: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
I20260809 00:58:58.020830 14757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } attempt: 4
I20260809 00:58:58.022342 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.021660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39718)
0809 00:58:58.021962 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:58.022097 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260809 00:58:58.023715 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.023214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39728)
0809 00:58:58.023456 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:58.023575 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 00:58:58.024025 14757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:46549: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
W20260809 00:58:58.082605 14774 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 00:58:58.083007 14774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 00:58:58.083079 14774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 00:58:58.083166 14774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 00:58:58.090821 14774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 00:58:58.090940 14774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 00:58:58.091003 14774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 00:58:58.091059 14774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 00:58:58.091123 14774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 00:58:58.101990 14774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.225.20:39183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.225.62:41643,127.9.225.61:43995,127.9.225.60:46549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/krb5kdc/kudu_127.9.225.60.keytab
--principal=kudu/127.9.225.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.225.60:46549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.225.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 23:55:54 UTC on ccycloud.quasar-bjxnky.root.comops.site
build id 4575725
ASAN enabled
I20260809 00:58:58.103421 14774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 00:58:58.104877 14774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 00:58:58.117880 14779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:58.117973 14783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 00:58:58.117928 14780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 00:58:58.119102 14774 server_base.cc:1034] running on GCE node
I20260809 00:58:58.120070 14774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 00:58:58.121850 14774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 00:58:58.123138 14774 hybrid_clock.cc:648] HybridClock initialized: now 1786237138123065 us; error 52 us; skew 500 ppm
I20260809 00:58:58.123572 14774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 00:58:58 dist-test-slave-j2vl krb5kdc[14517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237138, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 00:58:58.127178 14774 init.cc:377] Logged in from keytab as kudu/127.9.225.60@KRBTEST.COM (short username kudu)
I20260809 00:58:58.127887 14643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } attempt: 4
I20260809 00:58:58.129076 14774 webserver.cc:460] Webserver started at http://127.9.225.60:35917/ using document root <none> and password file <none>
I20260809 00:58:58.129180 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.128609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39732)
0809 00:58:58.128816 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:58.128976 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 00:58:58.130007 14774 fs_manager.cc:359] Metadata directory not provided
I20260809 00:58:58.130160 14774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 00:58:58.130357 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.129914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39734)
0809 00:58:58.130073 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:58.130185 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 00:58:58.130589 14774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
W20260809 00:58:58.130607 14643 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.225.60:46549: Network error: Client connection negotiation failed: client connection to 127.9.225.60:46549: connect: Connection refused (error 111)
I20260809 00:58:58.134166 14774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/data/instance:
uuid: "6dda0f5951dc40f98d02a7fc93211477"
format_stamp: "Formatted at 2026-08-09 00:58:58 on dist-test-slave-j2vl"
server_key: "464c4d4cf770286ea3e0eceae40e4c8b"
server_key_iv: "a7c1e8374bb64fe7ba85f0bed7358c4a"
server_key_version: "encryptionkey@0"
I20260809 00:58:58.135136 14774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/wal/instance:
uuid: "6dda0f5951dc40f98d02a7fc93211477"
format_stamp: "Formatted at 2026-08-09 00:58:58 on dist-test-slave-j2vl"
server_key: "464c4d4cf770286ea3e0eceae40e4c8b"
server_key_iv: "a7c1e8374bb64fe7ba85f0bed7358c4a"
server_key_version: "encryptionkey@0"
I20260809 00:58:58.141690 14774 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 00:58:58.145874 14797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 00:58:58.147457 14774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 00:58:58.147682 14774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/data,/tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/wal
uuid: "6dda0f5951dc40f98d02a7fc93211477"
format_stamp: "Formatted at 2026-08-09 00:58:58 on dist-test-slave-j2vl"
server_key: "464c4d4cf770286ea3e0eceae40e4c8b"
server_key_iv: "a7c1e8374bb64fe7ba85f0bed7358c4a"
server_key_version: "encryptionkey@0"
I20260809 00:58:58.147951 14774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 00:58:58.171052 14774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 00:58:58.176093 14774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 00:58:58.176381 14774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 00:58:58.198096 14774 rpc_server.cc:307] RPC server started. Bound to: 127.9.225.60:46549
I20260809 00:58:58.198114 14887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.225.60:46549 every 8 connection(s)
I20260809 00:58:58.199821 14774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/data/info.pb
I20260809 00:58:58.204056 10116 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskizkHom/build/asan/bin/kudu as pid 14774
I20260809 00:58:58.204273 10116 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskizkHom/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786236719337175-10116-0/minicluster-data/master-2/wal/instance
I20260809 00:58:58.204804 10116 external_mini_cluster.cc:1428] Setting key 6c666766dd5a024489cac6c0ce2466a1
I20260809 00:58:58.205061 14888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 00:58:58.217089 14888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:58:58 dist-test-slave-j2vl krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.60@KRBTEST.COM
Aug 09 00:58:58 dist-test-slave-j2vl krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237138, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.60@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
I20260809 00:58:58.233842 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.206998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.9.225.60:46549)
0809 00:58:58.208170 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.208189 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.208247 (+ 58us) server_negotiation.cc:407] Connection header received
0809 00:58:58.210153 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.210192 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.210897 (+ 705us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.211501 (+ 604us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.213276 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.215448 (+ 2172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.217004 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.218211 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.218437 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.224286 (+ 5849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:58.224298 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:58.224315 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:58.224372 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:58.228254 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.229437 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.229445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.229456 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.229662 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.230140 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.230144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.230146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.230815 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:58.231112 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:58.231701 (+ 589us) server_negotiation.cc:299] Negotiation successful
0809 00:58:58.232237 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":594,"threads_started":1}
I20260809 00:58:58.239450 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.220918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.62:41643 (local address 127.0.0.1:46294)
0809 00:58:58.221862 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:58.221908 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:58.222141 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:58.222628 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:58.222636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:58.223591 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:58.223991 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:58.224007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.225549 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.225553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:58.226961 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:58.226979 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.228046 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.228052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:58.228200 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.228897 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:58.228922 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:58.231695 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:58.235816 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:58.235826 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:58.235832 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:58.236391 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:58.236954 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:58.236967 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:58.236969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:58.237136 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:58.238061 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:58.238075 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:58.238328 (+ 253us) client_negotiation.cc:769] Sending connection context
0809 00:58:58.238567 (+ 239us) client_negotiation.cc:240] Negotiation successful
0809 00:58:58.238955 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":549,"threads_started":1}
I20260809 00:58:58.239543 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.220870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.9.225.62:41643)
0809 00:58:58.221470 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.221479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.222012 (+ 533us) server_negotiation.cc:407] Connection header received
0809 00:58:58.222321 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.222330 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.222415 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.222510 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.224183 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.225335 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.227153 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.227806 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.227968 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.231980 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:58.231988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:58.231994 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:58.232060 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:58.235629 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.236583 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.236589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.236593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.236802 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.237260 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.237265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.237269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.237732 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:58.237933 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:58.238672 (+ 739us) server_negotiation.cc:299] Negotiation successful
0809 00:58:58.238959 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":426,"thread_start_us":309,"threads_started":1}
I20260809 00:58:58.241191 14888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:58:58 dist-test-slave-j2vl krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.62@KRBTEST.COM
Aug 09 00:58:58 dist-test-slave-j2vl krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237138, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.60@KRBTEST.COM for kudu/127.9.225.61@KRBTEST.COM
I20260809 00:58:58.256135 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.237536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.9.225.62:41643)
0809 00:58:58.238191 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.238195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.238214 (+ 19us) server_negotiation.cc:407] Connection header received
0809 00:58:58.238253 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.238299 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.238385 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.238509 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.240210 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.241318 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.243228 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.243928 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.244021 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.249197 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:58.249203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:58.249206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:58.249249 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:58.252550 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.253632 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.253637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.253639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.253790 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.254292 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.254301 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.254307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.254770 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:58.255024 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:58.255501 (+ 477us) server_negotiation.cc:299] Negotiation successful
0809 00:58:58.255763 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":534,"thread_start_us":313,"threads_started":1}
I20260809 00:58:58.257442 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.242025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.61:43995 (local address 127.0.0.1:35882)
0809 00:58:58.242214 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:58.242238 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:58.242359 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:58.243008 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:58.243011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:58.243519 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:58.243708 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:58.243719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.245149 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.245153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:58.246124 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:58.246136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.247171 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.247176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:58.247324 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.247970 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:58.247996 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:58.250334 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:58.254095 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:58.254099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:58.254101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:58.254731 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:58.255293 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:58.255296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:58.255298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:58.255542 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:58.256334 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:58.256339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:58.256506 (+ 167us) client_negotiation.cc:769] Sending connection context
0809 00:58:58.256663 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 00:58:58.257007 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 00:58:58.257586 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.242118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.9.225.61:43995)
0809 00:58:58.242672 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.242679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.242698 (+ 19us) server_negotiation.cc:407] Connection header received
0809 00:58:58.242760 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.242765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.242850 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.242962 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.243906 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.244985 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.246296 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.247002 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.247157 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.250557 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:58.250562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:58.250565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:58.250601 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:58.253936 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.254955 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.254961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.254964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.255156 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.255684 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.255764 (+ 80us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.255768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.256188 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:58.256366 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:58.256787 (+ 421us) server_negotiation.cc:299] Negotiation successful
0809 00:58:58.257035 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":301,"threads_started":1}
I20260809 00:58:58.258889 14888 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 00:58:58 dist-test-slave-j2vl krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237138, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.60@KRBTEST.COM for kudu/127.9.225.60@KRBTEST.COM
Aug 09 00:58:58 dist-test-slave-j2vl krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.225.61@KRBTEST.COM
I20260809 00:58:58.274549 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.259670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39752)
0809 00:58:58.259853 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:58.259874 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:58.259985 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:58.260295 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:58.260299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:58.260885 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:58.261068 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:58.261080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.262301 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.262305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:58.263312 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:58.263327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.264108 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.264112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:58.264262 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.265031 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:58.265055 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:58.267725 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:58.271546 (+ 3821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:58.271553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:58.271558 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:58.272099 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:58.272535 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:58.272540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:58.272543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:58.272729 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:58.273368 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:58.273374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:58.273565 (+ 191us) client_negotiation.cc:769] Sending connection context
0809 00:58:58.273752 (+ 187us) client_negotiation.cc:240] Negotiation successful
0809 00:58:58.274087 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 00:58:58.274978 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.259700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.9.225.60:46549)
0809 00:58:58.259859 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.259866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.259924 (+ 58us) server_negotiation.cc:407] Connection header received
0809 00:58:58.260102 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.260107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.260178 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.260276 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.261221 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.262149 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.263462 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.263936 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.264070 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.267992 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:58.268000 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:58.268005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:58.268052 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:58.271370 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.272260 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.272264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.272267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.272405 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.272850 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.272853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.272855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.273186 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:58.273329 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:58.273892 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 00:58:58.274323 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":79}
I20260809 00:58:58.277719 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.258349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35888 (local address 127.9.225.61:43995)
0809 00:58:58.258668 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.258674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.258696 (+ 22us) server_negotiation.cc:407] Connection header received
0809 00:58:58.258773 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.258778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.258863 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.258973 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.260589 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.261652 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.263106 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.263701 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.263798 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.269499 (+ 5701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:58.269506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:58.269509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:58.269548 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:58.272810 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.273834 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.273841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.273845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.274043 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.274594 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.274624 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.274629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.275068 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:58.275364 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:58.275986 (+ 622us) server_negotiation.cc:299] Negotiation successful
0809 00:58:58.276296 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 00:58:58.281504 14888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477: Bootstrap starting.
I20260809 00:58:58.283891 10116 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 00:58:58.286165 14888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:58.287631 14888 log.cc:824] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:58.287372 14603 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2f7d1ec733a94886b93d7d89b68e2074"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46300
I20260809 00:58:58.291080 14888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477: No bootstrap required, opened a new log
I20260809 00:58:58.298877 14888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } }
I20260809 00:58:58.299309 14888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dda0f5951dc40f98d02a7fc93211477, State: Initialized, Role: FOLLOWER
I20260809 00:58:58.299814 14888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } }
I20260809 00:58:58.301016 14904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } } }
I20260809 00:58:58.301472 14904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:58.302202 14888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 00:58:58.313249 14919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 00:58:58.314533 14919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 00:58:58.310914 14774 master_runner.cc:186] Error getting master registration for 127.9.225.62:41643: OK, instance_id { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" instance_seqno: 1786237136973897 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 00:58:58.320428 14774 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 00:58:58.323292 14757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } attempt: 5
Aug 09 00:58:58 dist-test-slave-j2vl krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237137, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.61@KRBTEST.COM for kudu/127.9.225.60@KRBTEST.COM
I20260809 00:58:58.341847 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.324185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39758 (local address 127.9.225.60:46549)
0809 00:58:58.324417 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.324425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.324456 (+ 31us) server_negotiation.cc:407] Connection header received
0809 00:58:58.324565 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.324571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.324662 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.324808 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.325928 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.327155 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.328989 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.329802 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.329969 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.334175 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:58.334182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:58.334185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:58.334225 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:58.338081 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.339254 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.339261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.339265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.339492 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.340015 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.340021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.340025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.340502 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:58.340682 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:58.341134 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 00:58:58.341449 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 00:58:58.341881 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.324095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39758)
0809 00:58:58.324296 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:58.324317 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:58.324439 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:58.324873 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:58.324878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:58.325502 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:58.325728 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:58.325740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.327363 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.327369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:58.328801 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:58.328819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.329992 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.329999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:58.330251 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.331071 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:58.331096 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:58.333915 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:58.338309 (+ 4394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:58.338317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:58.338320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:58.339070 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:58.339661 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:58.339670 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:58.339675 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:58.339880 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:58.340647 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:58.340655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:58.340841 (+ 186us) client_negotiation.cc:769] Sending connection context
0809 00:58:58.341022 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 00:58:58.341457 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 00:58:58.349166 14757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717: Bootstrap starting.
I20260809 00:58:58.354244 14757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:58.355993 14757 log.cc:824] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:58.360368 14757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717: No bootstrap required, opened a new log
I20260809 00:58:58.372913 14757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } }
I20260809 00:58:58.373445 14757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48652feb3e97483ea89959893a6ce717, State: Initialized, Role: FOLLOWER
I20260809 00:58:58.374128 14757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } }
I20260809 00:58:58.375416 14923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } } }
I20260809 00:58:58.375901 14923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:58.377099 14757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:58.386752 14655 master_runner.cc:186] Error getting master registration for 127.9.225.62:41643: OK, instance_id { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" instance_seqno: 1786237136973897 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 00:58:58.399189 14655 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 00:58:58.402607 14940 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 00:58:58.402833 14940 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 00:58:58.407877 14643 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } attempt: 5
Aug 09 00:58:58 dist-test-slave-j2vl krb5kdc[14517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786237136, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.225.62@KRBTEST.COM for kudu/127.9.225.60@KRBTEST.COM
I20260809 00:58:58.425787 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.408728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.9.225.60:46549)
0809 00:58:58.409300 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.409304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.409320 (+ 16us) server_negotiation.cc:407] Connection header received
0809 00:58:58.409453 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.409460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.409574 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.409678 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.410572 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.411752 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.413029 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.413800 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.413930 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.418279 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:58.418284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:58.418288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:58.418325 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:58.422396 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.423293 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.423302 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.423306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.423521 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.424014 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.424020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.424023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.424416 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:58.424560 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:58.425113 (+ 553us) server_negotiation.cc:299] Negotiation successful
0809 00:58:58.425428 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445}
I20260809 00:58:58.425815 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.408663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.225.60:46549 (local address 127.0.0.1:39772)
0809 00:58:58.408956 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 00:58:58.408983 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 00:58:58.409092 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 00:58:58.409730 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 00:58:58.409738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 00:58:58.410211 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0809 00:58:58.410382 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:58.410391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.411911 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.411917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:58.412860 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 00:58:58.412873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.413965 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.413972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 00:58:58.414176 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.414903 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 00:58:58.414928 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 00:58:58.418021 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 00:58:58.422621 (+ 4600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:58.422629 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:58.422633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:58.423122 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:58.423659 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 00:58:58.423665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 00:58:58.423669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 00:58:58.423871 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 00:58:58.424640 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 00:58:58.424645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 00:58:58.424830 (+ 185us) client_negotiation.cc:769] Sending connection context
0809 00:58:58.424989 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 00:58:58.425331 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
W20260809 00:58:58.429111 14804 tablet.cc:2369] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:58:58.431802 14643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074: Bootstrap starting.
I20260809 00:58:58.436206 14643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074: Neither blocks nor log segments found. Creating new log.
I20260809 00:58:58.437897 14643 log.cc:824] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074: Log is configured to *not* fsync() on all Append() calls
I20260809 00:58:58.442317 14643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074: No bootstrap required, opened a new log
I20260809 00:58:58.450706 14643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } }
I20260809 00:58:58.451215 14643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f7d1ec733a94886b93d7d89b68e2074, State: Initialized, Role: FOLLOWER
I20260809 00:58:58.452143 14643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } }
I20260809 00:58:58.453903 14945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } } }
I20260809 00:58:58.454754 14945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:58.455379 14643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 00:58:58.455473 14603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 00:58:58.455688 14603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:58.461578 14603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } }
W20260809 00:58:58.462522 14573 tablet.cc:2369] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:58:58.466413 14716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f7d1ec733a94886b93d7d89b68e2074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "48652feb3e97483ea89959893a6ce717"
I20260809 00:58:58.467078 14716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:58.472808 14716 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f7d1ec733a94886b93d7d89b68e2074 in term 1.
I20260809 00:58:58.473934 14571 leader_election.cc:302] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f7d1ec733a94886b93d7d89b68e2074, 48652feb3e97483ea89959893a6ce717; no voters:
I20260809 00:58:58.475816 14844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f7d1ec733a94886b93d7d89b68e2074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6dda0f5951dc40f98d02a7fc93211477"
I20260809 00:58:58.476377 14844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [term 0 FOLLOWER]: Advancing to term 1
I20260809 00:58:58.481468 14844 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f7d1ec733a94886b93d7d89b68e2074 in term 1.
I20260809 00:58:58.483053 14603 leader_election.cc:288] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [CANDIDATE]: Term 1 election: Requested vote from peers 48652feb3e97483ea89959893a6ce717 (127.9.225.61:43995), 6dda0f5951dc40f98d02a7fc93211477 (127.9.225.60:46549)
I20260809 00:58:58.483923 14945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 00:58:58.498028 14945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [term 1 LEADER]: Becoming Leader. State: Replica: 2f7d1ec733a94886b93d7d89b68e2074, State: Running, Role: LEADER
I20260809 00:58:58.499197 14945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } }
I20260809 00:58:58.527359 14954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f7d1ec733a94886b93d7d89b68e2074. Latest consensus state: current_term: 1 leader_uuid: "2f7d1ec733a94886b93d7d89b68e2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } } }
I20260809 00:58:58.528070 14954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:58.529561 14973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 00:58:58.560362 14973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260809 00:58:58.573781 14687 tablet.cc:2369] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 00:58:58.581981 14844 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [term 1 FOLLOWER]: Refusing update from remote peer 2f7d1ec733a94886b93d7d89b68e2074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:58:58.583819 14945 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:58:58.590349 14716 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [term 1 FOLLOWER]: Refusing update from remote peer 2f7d1ec733a94886b93d7d89b68e2074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 00:58:58.600190 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.532514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.9.225.62:41643)
0809 00:58:58.532853 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.532862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.548465 (+ 15603us) server_negotiation.cc:407] Connection header received
0809 00:58:58.549207 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.549216 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.549370 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.549474 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.556318 (+ 6844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.557553 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.578274 (+ 20721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.579036 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.579181 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.589789 (+ 10608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:58.589798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:58.589804 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:58.589862 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:58.593994 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.595062 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.595069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.595073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.595259 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.598080 (+ 2821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.598088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.598092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.598540 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:58.598829 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:58.599494 (+ 665us) server_negotiation.cc:299] Negotiation successful
0809 00:58:58.599792 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 00:58:58.600708 14945 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 00:58:58.634618 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.531971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.9.225.61:43995)
0809 00:58:58.532364 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.532372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.578123 (+ 45751us) server_negotiation.cc:407] Connection header received
0809 00:58:58.584085 (+ 5962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.584093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.584204 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.584286 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.586460 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.587657 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.595818 (+ 8161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.596644 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.596760 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.607868 (+ 11108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:58.607877 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:58.607892 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:58.607957 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:58.612059 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.614383 (+ 2324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.614391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.614397 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.614595 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.624155 (+ 9560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.624166 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.624170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.624710 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:58.625034 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:58.633793 (+ 8759us) server_negotiation.cc:299] Negotiation successful
0809 00:58:58.634183 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260809 00:58:58.650303 14904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f7d1ec733a94886b93d7d89b68e2074. Latest consensus state: current_term: 1 leader_uuid: "2f7d1ec733a94886b93d7d89b68e2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } } }
I20260809 00:58:58.650830 14904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:58.654755 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:58.578701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.9.225.60:46549)
0809 00:58:58.578970 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.578979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.618710 (+ 39731us) server_negotiation.cc:407] Connection header received
0809 00:58:58.619054 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.619061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.619181 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.619269 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.620997 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.622155 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.625752 (+ 3597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:58.626527 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:58.626650 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:58.646192 (+ 19542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 00:58:58.646201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 00:58:58.646205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 00:58:58.646271 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0809 00:58:58.650564 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.651827 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.651834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.651837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.652021 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 00:58:58.652566 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 00:58:58.652572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 00:58:58.652576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 00:58:58.653020 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 00:58:58.653291 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:58.653994 (+ 703us) server_negotiation.cc:299] Negotiation successful
0809 00:58:58.654307 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 00:58:58.680714 14988 mvcc.cc:205] Tried to move back new op lower bound from 7316427319615959040 to 7316427319314857984. Current Snapshot: MvccSnapshot[applied={T|T < 7316427319615959040}]
I20260809 00:58:58.689757 14923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f7d1ec733a94886b93d7d89b68e2074. Latest consensus state: current_term: 1 leader_uuid: "2f7d1ec733a94886b93d7d89b68e2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } } }
I20260809 00:58:58.690342 14923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:58.702370 14954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f7d1ec733a94886b93d7d89b68e2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } } }
I20260809 00:58:58.703043 14954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:58.703859 14954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f7d1ec733a94886b93d7d89b68e2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } } }
I20260809 00:58:58.704234 14954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074 [sys.catalog]: This master's current role is: LEADER
I20260809 00:58:58.704706 14987 mvcc.cc:205] Tried to move back new op lower bound from 7316427319615959040 to 7316427319314857984. Current Snapshot: MvccSnapshot[applied={T|T < 7316427319615959040}]
I20260809 00:58:58.706105 14923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f7d1ec733a94886b93d7d89b68e2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } } }
I20260809 00:58:58.706444 14923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:58.714183 15003 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 00:58:58.715862 14904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f7d1ec733a94886b93d7d89b68e2074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7d1ec733a94886b93d7d89b68e2074" member_type: VOTER last_known_addr { host: "127.9.225.62" port: 41643 } } peers { permanent_uuid: "48652feb3e97483ea89959893a6ce717" member_type: VOTER last_known_addr { host: "127.9.225.61" port: 43995 } } peers { permanent_uuid: "6dda0f5951dc40f98d02a7fc93211477" member_type: VOTER last_known_addr { host: "127.9.225.60" port: 46549 } } }
I20260809 00:58:58.716205 14904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 00:58:58.719343 14973 catalog_manager.cc:1347] Generated new cluster ID: b5704e94e999416d842f9d06a19a82f8
I20260809 00:58:58.719491 14973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 00:58:58.741633 14973 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 00:58:58.743711 14973 catalog_manager.cc:1504] Loading token signing keys...
I20260809 00:58:58.762240 14973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f7d1ec733a94886b93d7d89b68e2074: Generated new TSK 0
I20260809 00:58:58.764482 14973 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 00:58:58.930891 14898 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:58.916068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.9.225.62:41643)
0809 00:58:58.916310 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.916318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.928370 (+ 12052us) server_negotiation.cc:407] Connection header received
0809 00:58:58.928999 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.929009 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.929126 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.929208 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.930585 (+ 1377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46318: BlockingRecv error: recv got EOF from 127.0.0.1:46318 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260809 00:58:58.933915 14901 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:58.924571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.9.225.61:43995)
0809 00:58:58.924833 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.924840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.931779 (+ 6939us) server_negotiation.cc:407] Connection header received
0809 00:58:58.932098 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.932105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.932210 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.932298 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.933633 (+ 1335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35910: BlockingRecv error: recv got EOF from 127.0.0.1:35910 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260809 00:58:58.938338 14895 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:58.935510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39778 (local address 127.9.225.60:46549)
0809 00:58:58.935734 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.935745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.936377 (+ 632us) server_negotiation.cc:407] Connection header received
0809 00:58:58.936664 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.936670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.936782 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.936874 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.938051 (+ 1177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39778: BlockingRecv error: recv got EOF from 127.0.0.1:39778 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260809 00:58:58.982106 14901 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:58.971451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.9.225.61:43995)
0809 00:58:58.971687 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:58.971694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:58.979919 (+ 8225us) server_negotiation.cc:407] Connection header received
0809 00:58:58.980291 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:58.980298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:58.980443 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:58.980546 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:58.981846 (+ 1300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35922: BlockingRecv error: recv got EOF from 127.0.0.1:35922 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260809 00:58:59.026661 14901 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:59.017069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.9.225.61:43995)
0809 00:58:59.017284 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.017294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.024377 (+ 7083us) server_negotiation.cc:407] Connection header received
0809 00:58:59.024934 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.024941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.025056 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.025131 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:59.026399 (+ 1268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35930: BlockingRecv error: recv got EOF from 127.0.0.1:35930 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260809 00:58:59.054379 14901 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:59.047418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.9.225.61:43995)
0809 00:58:59.047678 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.047688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.052407 (+ 4719us) server_negotiation.cc:407] Connection header received
0809 00:58:59.052679 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.052684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.052833 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.052963 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:59.054054 (+ 1091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35936: BlockingRecv error: recv got EOF from 127.0.0.1:35936 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260809 00:58:59.075196 14901 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:59.071230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.9.225.61:43995)
0809 00:58:59.071618 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.071626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.072425 (+ 799us) server_negotiation.cc:407] Connection header received
0809 00:58:59.072988 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.073002 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.073178 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.073389 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:59.074915 (+ 1526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35946: BlockingRecv error: recv got EOF from 127.0.0.1:35946 (error 108)
Metrics: {"server-negotiator.queue_time_us":183}
W20260809 00:58:59.102998 14901 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:59.095760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.9.225.61:43995)
0809 00:58:59.095977 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.095984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.100916 (+ 4932us) server_negotiation.cc:407] Connection header received
0809 00:58:59.101159 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.101165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.101452 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.101597 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:59.102668 (+ 1071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35962: BlockingRecv error: recv got EOF from 127.0.0.1:35962 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260809 00:58:59.125692 14901 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:59.123095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.9.225.61:43995)
0809 00:58:59.123317 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.123324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.123920 (+ 596us) server_negotiation.cc:407] Connection header received
0809 00:58:59.124124 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.124130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.124253 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.124390 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:59.125411 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35972: BlockingRecv error: recv got EOF from 127.0.0.1:35972 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260809 00:58:59.164527 14901 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:59.161803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.9.225.61:43995)
0809 00:58:59.162053 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.162061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.162722 (+ 661us) server_negotiation.cc:407] Connection header received
0809 00:58:59.162957 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.162964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.163114 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.163252 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:59.164237 (+ 985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35976: BlockingRecv error: recv got EOF from 127.0.0.1:35976 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260809 00:58:59.234513 14901 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:59.232314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35990 (local address 127.9.225.61:43995)
0809 00:58:59.232523 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.232530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.232893 (+ 363us) server_negotiation.cc:407] Connection header received
0809 00:58:59.233099 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.233107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.233238 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.233425 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:59.234261 (+ 836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35990: BlockingRecv error: recv got EOF from 127.0.0.1:35990 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 00:58:59.316777 14919 catalog_manager.cc:1259] Loaded cluster ID: b5704e94e999416d842f9d06a19a82f8
I20260809 00:58:59.316936 14919 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477: loading cluster ID for follower catalog manager: success
I20260809 00:58:59.319840 14919 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477: acquiring CA information for follower catalog manager: success
I20260809 00:58:59.322018 14919 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6dda0f5951dc40f98d02a7fc93211477: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 00:58:59.368551 14901 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 00:58:59.366540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.9.225.61:43995)
0809 00:58:59.366800 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.366808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.367044 (+ 236us) server_negotiation.cc:407] Connection header received
0809 00:58:59.367237 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.367241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.367323 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.367430 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 00:58:59.368321 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35992: BlockingRecv error: recv got EOF from 127.0.0.1:35992 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260809 00:58:59.406041 14940 catalog_manager.cc:1259] Loaded cluster ID: b5704e94e999416d842f9d06a19a82f8
I20260809 00:58:59.406164 14940 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717: loading cluster ID for follower catalog manager: success
I20260809 00:58:59.409130 14940 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717: acquiring CA information for follower catalog manager: success
I20260809 00:58:59.411345 14940 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 48652feb3e97483ea89959893a6ce717: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 00:58:59.635268 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:59.628855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.9.225.61:43995)
0809 00:58:59.629123 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.629130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.629383 (+ 253us) server_negotiation.cc:407] Connection header received
0809 00:58:59.629641 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.629646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.629837 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.629973 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:58:59.630453 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:59.631194 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:59.632621 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:59.633052 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:59.633216 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:59.633875 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:58:59.634507 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:58:59.634692 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:59.634848 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 00:58:59.634951 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260809 00:58:59.648955 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:59.642363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.9.225.60:46549)
0809 00:58:59.643053 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.643059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.643078 (+ 19us) server_negotiation.cc:407] Connection header received
0809 00:58:59.643189 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.643192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.643366 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.643533 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:58:59.644023 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:59.644820 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:59.646393 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:59.646848 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:59.646973 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:59.647594 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:58:59.648277 (+ 683us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:58:59.648397 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:59.648590 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 00:58:59.648680 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":462,"threads_started":1}
I20260809 00:58:59.665400 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:59.656641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.9.225.61:43995)
0809 00:58:59.656896 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.656906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.658632 (+ 1726us) server_negotiation.cc:407] Connection header received
0809 00:58:59.658986 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.658996 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.659125 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.659205 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:58:59.660253 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:59.661628 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:59.663404 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:59.663868 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:59.664011 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:59.664500 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:58:59.664687 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:58:59.664783 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:59.664949 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 00:58:59.665049 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 00:58:59.666898 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:59.656654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.9.225.62:41643)
0809 00:58:59.658078 (+ 1424us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.658083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.658101 (+ 18us) server_negotiation.cc:407] Connection header received
0809 00:58:59.658216 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.658220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.658405 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.659748 (+ 1343us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:58:59.659912 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:59.661319 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:59.663821 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:59.664488 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:59.664624 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:59.665273 (+ 649us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:58:59.665853 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:58:59.666383 (+ 530us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:59.666442 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 00:58:59.666542 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":1226,"threads_started":1}
I20260809 00:58:59.667582 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 00:58:59.659114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.9.225.60:46549)
0809 00:58:59.659310 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 00:58:59.659316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 00:58:59.660848 (+ 1532us) server_negotiation.cc:407] Connection header received
0809 00:58:59.661050 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 00:58:59.661054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 00:58:59.661193 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 00:58:59.661306 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 00:58:59.662159 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:59.663256 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:59.665461 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 00:58:59.666121 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 00:58:59.666254 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 00:58:59.666771 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 00:58:59.666908 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 00:58:59.666981 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 00:58:59.667198 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 00:58:59.667287 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 00:58:59.672631 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 14534
I20260809 00:58:59.673976 14633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:59.790284 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 14534
W20260809 00:58:59.826874 14685 connection.cc:535] server connection from 127.0.0.1:35868 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 00:58:59.827935 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 14655
I20260809 00:58:59.829205 14749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:58:59.944118 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 14655
I20260809 00:58:59.978943 10116 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskizkHom/build/asan/bin/kudu pid 14774
I20260809 00:58:59.980321 14883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 00:59:00.092653 10116 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskizkHom/build/asan/bin/kudu with pid 14774
2026-08-09T00:59:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5462 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10859 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (420642 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST